Automated Evaluation

Follow this topic Ask about this topic

Automated evaluation studies how to measure the quality, correctness, safety, and usefulness of AI outputs and systems without relying entirely on manual review. Common approaches include reference-based metrics, reference-free scoring, LLM-as-a-judge, rubric and checklist scoring, pairwise comparisons, and executable tests that verify code, tools, or interactive behavior. Benchmarks increasingly assess agents through full task trajectories and evidence from browsers, databases, or other environments, while statistical calibration and targeted human review help interpret results. The central challenge is making automated scores valid, robust across tasks and populations, and resistant to bias or gaming; agreement with human ratings alone does not guarantee that an evaluator measures the intended capability.

Latest papers

398 papers38 papers in the last four weeks, down 16% on the four weeks before. 0.2% of all new papers.

A Novel Approach to Evaluating Evaluation Metrics for Multi-Output Structured Prediction
Oct 29, 2026Akshay Vyas, Angelo Pimienta, Nicholas RuozziAutomated Evaluation
The University of Texas at Dallas
Human Evaluation with Automatic Scores: Separating Sampling and Estimation Gains
Oct 28, 2026Yang Liu, Ran Zou, Zhengtao Gui +1Human-in-the-Loop Evaluation LLM Evaluation
Department of Statistics and Operations Research, University of North Carolina at Chapel Hill, USA · Department of Statistics, University of California, Irvine, USA · School of Computer Science and Engineering, University of New South Wales, Australia
Right Click, Wrong Box: Grading GUI Grounding by What a Click Does
Oct 27, 2026Xiaokang Yuan, Binhui Yao, Luwei Yang +2GUI Grounding Automated Evaluation
Zhejiang University, Hangzhou, China · Zhejiang Geely Holding Group Co., Ltd, Hangzhou, China
MAREval: A Multi-Agent Framework for Evaluating Natural Language Recommendation Explanations
Oct 26, 2026Reza Yousefi Maragheh, Jayesh Uddhav Kudase, Aysenur Inan +7Automated Evaluation LLM-as-a-Judge
Walmart Global Tech, Sunnyvale, CA, USA
From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing
Oct 26, 2026Harine Choi, Chenchen Kuai, Zihao Li +5LLM Evaluation LLM Auditing
Texas A&M Transportation Institute, Texas A&M University System · Zachry Department of Civil and Environmental Engineering, Texas A&M University · Department of Civil, Construction and Environmental Engineering, Marquette University +2
MechAlign: Evaluating Semantic Alignment at the Project Level in Mathematical Formalization
Oct 26, 2026Hao Shen, Ruichen Qiu, Lihong Zhi +1Semantic Alignment Automated Evaluation
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, and University of Chinese Academy of Sciences.
Distilling Expert Judgment at Scale
Oct 23, 2026Robbie Goldfarb, Andrew B. Hall, Jillian Fisher +2Automated Evaluation LLM-as-a-Judge
Forum AI · Stanford University · University of Washington
MQAScore: Fine-Grained Text-Music Alignment Evaluation via Large Audio Language Model
Oct 22, 2026Wei-Jaw Lee, Chun-Ping Wang, Xuanjun Chen +1Audio-Text Alignment Automated Evaluation
GICE, National Taiwan University · AI-CoRE, National Taiwan University · Taiwan AILabs
VisCurate: Evaluating Behavioral Equivalence of Visual Skills
Oct 21, 2026Md Farhan Ishraq, Md Farhan Ishmam, Reaz Hassan Joarder +1Automated Evaluation
Islamic University of Technology · University of Utah
SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges
Oct 20, 2026Ethan Ludwick, Yu-Chung HsiaoLLM-as-a-Judge Automated Test Generation
Cisco Systems · Anduril Industries
EasyResearch: Honest Regression, Forecasting and Signal Classification for Researchers Who Do Not Program
Oct 18, 2026Ahmad B. A. Hassanat, Ghada A. AltarawnehAutomated Evaluation Data Leakage
Faculty of Information Technology, Mutah University, Karak, Jordan · Department of Accounting, Mutah University, Karak, Jordan
The Collection of a Real-World, Large-Scale Dataset for Automated Skill Evaluation in Central Line Dressing Change
Oct 16, 2026Ummey Tanin, Shannan K. Hamlin, Hsin-Mei Chen +2Automated Evaluation Educational Technology
Pam · University of Houston Houston, TX, USA
QuAD: Quantitative Assessment via Distribution-Alignment for Qualitative Data Quality Enabled by a Content-Aware Multi-Agent System
Oct 16, 2026Wenshuo Zhang, Leixian Shen, Qiqi Zhou +6Automated Evaluation Distribution Matching
The Hong Kong University of Science and Technology · Shanghai Jiao Tong University · The Hong Kong University of Science and Technology (Guangzhou)
Null Controls Are Necessary but Not Sufficient: Evaluating Acceptance Guards for Scientific Discovery and Interpretability
Oct 15, 2026Tianyu WuScientific Discovery Automated Evaluation
Project Aether
EvalBuilder: Can LLMs Build Efficient Benchmarks?
Oct 14, 2026Huaisheng Zhu, Hangfan Zhang, Teng Xiao +5Benchmark Construction AI Agent Benchmarks
Meituan · Pennsylvania State University · Allen Institute for AI (AI2) +2
Assessing the Quality of Concrete Scenarios Generated from Scenario Classes for Automated Driving
Oct 13, 2026Lina Putze, Thies de Graaff, Jan Steffen Becker +1Autonomous Driving Scenario Generation Automated Evaluation
German Aerospace Center (DLR) e.V. Institute of Systems Engineering for Future Mobility, Oldenburg, Germany
One-Sided Verification Under Limited Expert Review
Oct 13, 2026Fan Nie, Haotian Ye, Shurui Liu +3Reasoning Verification Human-in-the-Loop Evaluation
Stanford University
FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval
Oct 13, 2026Tripti Shukla, Shwetha Somasundaram, Akshat Gupta +3Automated Evaluation IR Evaluation
Glance Bengaluru, India · Glance Vilnius, Lithuania · Glance London, United Kingdom
A Node Embedding Framework for Model-Centric Conformance Checking in Process Mining
Oct 13, 2026Mostafa Abbasi, Homayoun NajjaranProcess Mining Automated Evaluation
Faculty of Engineering and Computer Science, University of Victoria, Victoria, BC, Canada
Improving the Evaluation of Biomedical Question Answering with Few Human Preference Signals
Oct 12, 2026Shakila Mahjabin Tonni, Xiang Dai, Sarvnaz Karimi +3RAG Evaluation Automated Evaluation
Suncorp Bank, Australia · CSIRO, Australia · Monash University, Australia +1
Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes
Oct 8, 2026Utku Boran Torun, Veli Karakaya, Eray TüzünAutomated Evaluation LLM Agent Reliability
Bilkent University, Ankara, Turkey
Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents
Oct 8, 2026Xing Zhang, Guanghui Wang, Yanwei Cui +4Automated Evaluation LLM Agent Self-Improvement
AWS Forward Deployed Engineering · HSBC Holdings Plc., HSBC Technology Center, China
LLM Persuasion Is in the Eye of the Evaluation
Oct 7, 2026Kamile Dementaviciute, Julija Vaitonyte, Tijl De BiePersuasion in Language Models LLM Evaluation
Ghent University · Tilburg University, Department of Computational Cognitive Science · ISM University of Management and Economics
Perceptually Aligned Evaluation of Style Transfer
Oct 7, 2026Yang Deng, Eleftherios Ioannou, David Mould +3Image Style Transfer Automated Evaluation
Cardiff University, UK · The Cyprus Institute, Cyprus · Carleton University, Canada +1
A Tale of Two Error Categories: Exploring Concealed Trade-Offs in the Errors of Automated Judges in Evaluation of Uncertainty Quantifiers
Oct 7, 2026Evgenia Ilia, Wilker AzizUncertainty Quantification Automated Evaluation
University of Amsterdam
From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback
Oct 7, 2026Prabhath Chellingi, Raviraja G, Viraj BagalTaxonomy Induction Automated Evaluation
Enterpret
Language Carries the Expert's Impression: Instrument-Anchored LLM Judges Transfer Counseling-Quality Assessment and Beat In-Domain Training
Oct 6, 2026Tobias Hallmen, Elisabeth AndréMental Health Counseling Cross-Domain Transfer Learning
Chair for Human-Centered Artificial Intelligence University of Augsburg
CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets
Oct 5, 2026Berke Arda, Ahmetcan Yavuz, Paul Gerry +7Document Information Extraction LLM Information Extraction
ETH Zurich · CSAIL, MIT · Helmholtz Zentrum München +5
An Information-Theoretic Evaluation Framework for Benchmark and Model Diagnosis in Knowledge Tracing
Oct 4, 2026Houru Jiang, Zixi Wang, Tengteng Cheng +6Knowledge Tracing Automated Evaluation
Jinan University, Guangzhou, China · Guangdong Institute of Smart Education, Jinan University, Guangzhou, China · Jilin University, Changchun, China
AI-Enabled Quality Assurance for Multiple-Choice Assessment Items
Oct 3, 2026Steven Moore, Nicholas DianaAutomated Evaluation Educational Assessment
George Mason University · Colgate University

Oct 29, 2026J/K move · Enter open · S save

A Novel Approach to Evaluating Evaluation Metrics for Multi-Output Structured Prediction

Akshay Vyas, Angelo Pimienta, Nicholas Ruozzi

The University of Texas at Dallas

Automated Evaluation

Open paper AskPDF arXiv

In multi-output structured prediction tasks, while only one ground truth label may be provided in the training data, multiple equally valid outputs may be possible, making reliable evaluation a persistent challenge. We postulate that human evaluators implicitly use task-specific invariants, e.g., object boundaries in colorized images or named entities in translations, to judge if an output is acceptable. Under this assumption, we introduce a notion of approximate task-specific invariants and use them as diagnostic tools to evaluate a variety of existing metrics for vision and language tasks. We use these task invariants as part of a framework to systematically test metric reliability by encouraging domain-relevant invariants in model outputs via an augmented loss function. In our experiments, we observe that enforcing invariants with an augmented loss yields substantial improvements in popular distributional metrics while more traditional metrics change only marginally. Through this invariants-driven evaluation, we expose where standard metrics fail to detect meaningful differences, and we highlight the conditions under which distributional metrics succeed or still fall short.