Automated Evaluation

Follow this topic Ask about this topic

Automated evaluation studies how to measure the quality, correctness, safety, and usefulness of AI outputs and systems without relying entirely on manual review. Common approaches include reference-based metrics, reference-free scoring, LLM-as-a-judge, rubric and checklist scoring, pairwise comparisons, and executable tests that verify code, tools, or interactive behavior. Benchmarks increasingly assess agents through full task trajectories and evidence from browsers, databases, or other environments, while statistical calibration and targeted human review help interpret results. The central challenge is making automated scores valid, robust across tasks and populations, and resistant to bias or gaming; agreement with human ratings alone does not guarantee that an evaluator measures the intended capability.

Latest papers

398 papers38 papers in the last four weeks, down 16% on the four weeks before. 0.2% of all new papers.

What Makes a Good Response? An Empirical Analysis of Quality in Qualitative Interviews
Apr 6, 2026Jonathan Ivey, Anjalie Field, Ziang XiaoAutomated Evaluation
Johns Hopkins University
Adaptive Cost-Efficient Evaluation for Reliable Patent Claim Generation
Apr 5, 2026Yongmin Yoo, Qiongkai Xu, Longbing CaoClaim Verification Automated Evaluation
Frontier AI Research Centre, Macquarie University · School of Computing, FSE, Macquarie University
Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean
Apr 4, 2026Aleksandr MeshkovAutomated Evaluation Language Model Generation Evaluation
Independent Researcher, Budva, Montenegro.
The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost
Apr 3, 2026Scott FrohnMathematics Education Automated Evaluation
Khan Academy, Mountain View, CA, USA
Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite
Apr 2, 2026Klaudia Thellmann, Bernhard Stadler, Michael FärberBenchmark Auditing MT Evaluation
TU Dresden and ScaDS.AI · InfAI e.V.
REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage
Apr 2, 2026Smriti Jha, Matteo Paltenghi, Chandra Maddila +3Coding Agent Benchmarks Benchmark Construction
Meta, USA
When VLMs 'Fix' Students: Identifying and Penalizing Over-Correction in the Evaluation of Multi-line Handwritten Math OCR
Apr 1, 2026Jin Seong, Wencke Liermann, Minho Kim +2Handwritten Mathematical Expression Recognition Self-Correction in VLMs
Electronics and Telecommunications Research Institute, Republic of Korea
SEA: Evaluating Sketch Abstraction Efficiency via Element-level Commonsense Visual Question Answering
Mar 30, 2026Jiho Park, Sieun Choi, Jaeyoon Seo +3Automated Evaluation VLM Evaluation
Dongguk University, Republic of Korea
Learning in Blocks: A Multi Agent Debate Assisted Personalized Adaptive Learning Framework for Language Learning
Mar 29, 2026Nicy Scaria, Silvester John Joseph Kennedy, Deepak SubramaniMulti-Agent Debate Automated Evaluation
Indian Institute of Science, India · Talking Yak, India · Indian Institute of Technology Patna, India
CREval: An Automated Interpretable Evaluation for Creative Image Manipulation under Complex Instructions
Mar 27, 2026Chonghuinan Wang, Zihan Chen, Yuxiang Wei +5Image Editing Evaluation Automated Evaluation
Harbin Institute of Technology · Huawei Noah’s Ark Lab · Pengcheng Lab, Guangzhou
When AI Meets Early Childhood Education: Large Language Models as Assessment Teammates in Chinese Preschools
Mar 25, 2026Xingming Li, Runke Huang, Yanan Bao +3Educational Assessment Automated Evaluation
National University of Defense Technology, Changsha, China · The Chinese University of Hong Kong, Shenzhen, China · University of Oxford, Oxford, United Kingdom
Evaluating the Reliability and Fidelity of Automated Judgment Systems of Large Language Models
Mar 23, 2026Tom Biskupski, Stephan KleberLLM-as-a-Judge Automated Evaluation
Hochschule der Medien Stuttgart, Germany · Ravensburg-Weingarten University of Applied Sciences Weingarten, Germany
Beyond String Matching: Semantic Evaluation of PDF Table Extraction
Mar 19, 2026Pius Horn, Janis KeuperDocument Parsing Benchmarks Document Information Extraction
Institute for Machine Learning and Analytics (IMLA), Offenburg University, Offenburg, Germany · University of Mannheim, Mannheim, Germany
CoMAI: A Collaborative Multi-Agent Framework for Robust and Equitable Interview Evaluation
Mar 17, 2026Gengxin Sun, Ruihao Yu, Liangyi Yin +3Automated Evaluation Multi-Agent Systems
Shandong University, Qingdao, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Shandong University, Jinan, China
Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis
Mar 16, 2026Penny Chong, Harshavardhan Abichandani, Jiyuan Shen +4LLM Agent Evaluation Agent Evaluation
SAP · Stanford University
Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI
Mar 16, 2026Jinhu Qi, Yifan Li, Minghao Zhao +4AI Agent Evaluation AI Agent Safety
The Chinese University of Hong Kong, Hong Kong SAR, China · Macao Polytechnic University, Macao SAR, China · Jilin University, Changchun, China
Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications
Mar 13, 2026Alexandre Cristovão MaioranoAutomated Evaluation
AnimeScore: A Preference-Based Dataset and Framework for Evaluating Anime-Like Speech Style
Mar 12, 2026Joonyong Park, Jerry LiAutomated Evaluation Speech Quality Assessment
Spellbrush, USA
ECoLAD: Selecting Anomaly Detectors for Automotive Deployment via Compute-Reduction Evaluation
Mar 11, 2026Kadir-Kaan Özer, René Ebeling, Markus EnzweilerEfficient Neural Network Inference Automated Evaluation
Mercedes-Benz AG, Germany. · Esslingen University of Applied Sciences, Germany.
End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering
Mar 11, 2026Nhi Dang, Tung Le, Huy Tien NguyenAutomated Evaluation RAG Evaluation
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam; Vietnam National University, Ho Chi Minh City, Vietnam
Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas
Mar 11, 2026Tim Schopf, Michael FärberAutomated Evaluation LLM-as-a-Judge
TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan
One-Eval: An Agentic System for Automated and Traceable LLM Evaluation
Mar 10, 2026Chengyu Shen, Yanheng Hou, Minghui Pan +8Automated Evaluation LLM Evaluation
Peking University · Beijing Institute of Technology · Beijing University of Posts and Telecommunications +1
MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistants
Mar 10, 2026Zuhao Zhang, Chengyue Yu, Yuante Li +3Web Application Generation LLM Evaluation
Inclusion AI, Ant Group, Hangzhou, Zhejiang, China · Shanghai Jiao Tong University, Shanghai, China · Carnegie Mellon University, Pittsburgh, PA, USA
Evaluating LLM-Based Grant Proposal Review via Structured Perturbations
Mar 9, 2026William Thorne, Joseph James, Yang Wang +2Automated Peer Review Automated Evaluation
University of Sheffield, UK · University of Manchester, UK
Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
Mar 9, 2026Hongli Zhou, Hui Huang, Rui Zhang +5LLM-as-a-Judge Social Bias in Language Models
Faculty of Computing, Harbin Institute of Technology, Harbin, 150001, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518055, China
An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data
Mar 8, 2026Trinh Pham, Thanh Tam Nguyen, Viet Huynh +2Automated Evaluation Text-to-SQL
Griffith University (Australia) · Edith Cowan University (Australia) · The University of Queensland (Australia)
Beyond Surrogates: A Quantitative Analysis for Inter-Metric Relationships
Mar 8, 2026Yuanhao Pu, Defu Lian, Enhong ChenAutomated Evaluation
School of Artificial Intelligence & Data Science, University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence, China · School of Computer Science & Technology, University of Science and Technology of China
PresentBench: A Fine-Grained Rubric-Based Benchmark for Slide Generation
Mar 7, 2026Xin-Sheng Chen, Jiayu Zhu, Pei-lin Li +3Benchmark Design Generative AI Evaluation
Tsinghua University
AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge
Mar 7, 2026Karen Zhou, Chenhao TanAutomated Evaluation Rubric-Based Evaluation
University of Chicago
Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks
Mar 6, 2026Jena D. Hwang, Varsha Kishore, Amanpreet Singh +9Automated Evaluation RAG Evaluation
Allen Institute for AI

Apr 6, 2026J/K move · Enter open · S save

What Makes a Good Response? An Empirical Analysis of Quality in Qualitative Interviews

Jonathan Ivey, Anjalie Field, Ziang Xiao

Johns Hopkins University

Automated Evaluation

Open paper AskPDF arXiv

Qualitative interviews provide essential insights into human experiences when they elicit high-quality responses. While qualitative and NLP researchers have proposed various measures of interview quality, these measures lack validation that high-scoring responses actually contribute to the study's goals. In this work, we identify, implement, and evaluate 10 proposed measures of interview response quality to determine which are actually predictive of a response's contribution to the study findings. To conduct our analysis, we introduce the Qualitative Interview Corpus, a newly constructed dataset of 343 interview transcripts with 16,940 participant responses from 14 real research projects. We find that direct relevance to a key research question is the strongest predictor of response quality. We additionally find that two measures commonly used to evaluate NLP interview systems, clarity and surprisal-based informativeness, are not predictive of response quality. Our work provides analytic insights and grounded, scalable metrics to inform the design of qualitative studies and the evaluation of automated interview systems.