Automated Evaluation

Follow this topic Ask about this topic

Automated evaluation studies how to measure the quality, correctness, safety, and usefulness of AI outputs and systems without relying entirely on manual review. Common approaches include reference-based metrics, reference-free scoring, LLM-as-a-judge, rubric and checklist scoring, pairwise comparisons, and executable tests that verify code, tools, or interactive behavior. Benchmarks increasingly assess agents through full task trajectories and evidence from browsers, databases, or other environments, while statistical calibration and targeted human review help interpret results. The central challenge is making automated scores valid, robust across tasks and populations, and resistant to bias or gaming; agreement with human ratings alone does not guarantee that an evaluator measures the intended capability.

Latest papers

398 papers38 papers in the last four weeks, down 16% on the four weeks before. 0.2% of all new papers.

Honest and Reliable Evaluation and Expert Equivalence Testing of Automated Neonatal Seizure Detection
Aug 6, 2025Jovana Kljajic, John M. O'Toole, Robert Hogan +1Inter-Rater Reliability Automated Evaluation
Faculty of Technical Sciences, University of Novi Sad, 21000 Novi Sad, Serbia · CergenX Ltd, Dublin, Ireland
The NTNU System at the S&I Challenge 2025 SLA Open Track
Jun 5, 2025Hong-Yun Lin, Tien-Hong Lo, Yu-Hsuan Fang +4Audio-Language Models Automated Evaluation
Department of Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · Institute of AI Interdisciplinary Applied Technology, National Taiwan Normal University, Taiwan
Multimodal Language Models as Text-to-Image Model Evaluators
May 1, 2025Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5LLM-as-a-Judge Benchmark Design
1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta +3
SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA
Apr 10, 2025Sher Badshah, Ali Emami, Hassan SajjadLLM Evaluation Language Model Generation Evaluation
Dalhousie University · Emory University
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
Mar 21, 2025Jeremy Barnes, Naiara Perez, Alba Bonet-Jover +1Summarization Evaluation Automated Evaluation
HiTZ Center, University of the Basque Country (UPV/EHU) · Department of Software and Computing Systems, University of Alicante, Spain · GOI Institute, Basque Summer University (UEU)
Cohort-attention Evaluation Metrics for Tied Data
Mar 17, 2025Dongjing Jiang, Qingchong JiaoAlgorithmic Fairness Automated Evaluation
DeepCox, Canada
A Course Intelligence Platform for Higher Education: Lessons from AI-Assisted Course Evaluation
Nov 3, 2024Bo Yuan, Jiazi Hu, Haimei ZhaoEducational Assessment Generative AI in Education
University of Queensland, Brisbane, QLD 4072, Australia · AI U Course, Kunming, Yunnan 650000, P.R. China
GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models
Dec 11, 2023Jiaxu Zhao, Meng Fang, Shirui Pan +2Language Model Bias Evaluation Automated Evaluation
Mathematics and Computer Science · Eindhoven University of Technology · Eindhoven, Netherlands +9

Aug 6, 2025J/K move · Enter open · S save

Honest and Reliable Evaluation and Expert Equivalence Testing of Automated Neonatal Seizure Detection

Jovana Kljajic, John M. O'Toole, Robert Hogan, Tamara Skoric

Faculty of Technical Sciences, University of Novi Sad, 21000 Novi Sad, Serbia · CergenX Ltd, Dublin, Ireland

Inter-Rater Reliability Automated Evaluation

Open paper AskPDF arXiv

Reliable evaluation of machine learning models for neonatal seizure detection is critical for clinical adoption. Current practices often rely on inconsistent and biased metrics, hindering model comparability and interpretability. Expert-level claims about AI performance are frequently made without rigorous validation, raising concerns about their reliability. This study aims to systematically evaluate common performance metrics and propose best practices tailored to the specific challenges of neonatal seizure detection. Using real and synthetic seizure annotations, we assessed standard performance metrics, consensus strategies, and human-expert level equivalence tests under varying class imbalance, inter-rater agreement, and number of raters. Matthews and Pearson's correlation coefficients outperformed the area under the curve in reflecting performance under class imbalance. Consensus types are sensitive to the number of raters and agreement level among them. Among human-expert level equivalence tests, the multi-rater Turing test using Fleiss k best captured expert-level AI performance. We recommend reporting: (1) at least one balanced metric, (2) Sensitivity, specificity, PPV and NPV, (3) Multi-rater Turing test results using Fleiss k, and (4) All the above on held-out validation set. This proposed framework provides an important prerequisite to clinical validation by enabling a thorough and honest appraisal of AI methods for neonatal seizure detection.