Overview

Automated evaluation studies how to measure the quality, correctness, safety, and usefulness of AI outputs and systems without relying entirely on manual review. Common approaches include reference-based metrics, reference-free scoring, LLM-as-a-judge, rubric and checklist scoring, pairwise comparisons, and executable tests that verify code, tools, or interactive behavior. Benchmarks increasingly assess agents through full task trajectories and evidence from browsers, databases, or other environments, while statistical calibration and targeted human review help interpret results. The central challenge is making automated scores valid, robust across tasks and populations, and resistant to bias or gaming; agreement with human ratings alone does not guarantee that an evaluator measures the intended capability.

Latest papers

CardsList

422 papers28 papers in the last four weeks, down 35% on the four weeks before. 0.2% of all new papers.

  1. Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks

    Nov 18, 2026Zirui Song, Jiadong Zhang, Zheng Lu +3Automated EvaluationRobot Manipulation Benchmarks

  2. Prioritize, Don't Decide: What We Learned Auditing a Deployed Human-in-the-Loop CV Sorter

    Nov 17, 2026Hadi Mohammadi, Jeroen Sparla, Alexandros Paliouras +2Human-in-the-Loop EvaluationLearning to Rank

  3. Beyond Acuity Accuracy: A Multidimensional Framework for Evaluating Patient-Facing Emergency Triage AI Tools

    Nov 17, 2026Ruio Heerschap, Henry Foote, Mahmoud Alwakeel +19Clinical TriageClinical Language Model Evaluation

  4. ParseBench-2: Severity-Aware Evaluation of Document Parsing

    Nov 12, 2026Pierre-Loïc Doulcet, Boyang Zhang, Sebastián G. Acosta +5Document Parsing BenchmarksDocument Parsing

  5. Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline

    Nov 12, 2026Geoffrey Challen, Craig ZillesEducationAutomated Evaluation

  6. SAGE-QA: Semantic-Aware, Grounded Evaluation for Closed-Context Question Answering on SQuAD 2.0

    Nov 12, 2026Chandra N Sekharan, Vishakan Umapathy, Abhishek PhadkeLLM EvaluationAutomated Evaluation

  7. Verifier Choice is a Benchmark Design Variable: Auditing Structural Counting Evaluation in Text-to-Image Models

    Nov 11, 2026Shurun Li, Michael Hua Wang, Haibo Zeng +1Automated EvaluationT2I Generation Evaluation

  8. Pterodactyl: Pushing the Limits of Efficient Automated Translation Evaluation

    Nov 10, 2026Evelyn Y. Garland, Carola F. BergerMT EvaluationAutomated Evaluation

  9. Assessing Alignment in Automatic Piano Transcription Datasets: A Benchmark for Scoring Methods

    Nov 10, 2026Lucas de Souza Brandão, Lucas Gris, Joaquim Cavalcante +2Automated EvaluationAutomatic Music Transcription

  10. Ninety-Six Ways to Measure a Scribe: Fragmented Evaluation of Ambient Scribes in Healthcare

    Nov 10, 2026Amama Mahmood, Shiye Cao, Laura Fleig +5Automated Evaluation

  11. SAUCE: Summary Analysis Using Conversation Entailment

    Nov 4, 2026Man-Ling Sung, Hemanth Kandula, Jeff Ma +2Summarization EvaluationAutomated Evaluation

  12. Ranking Agreement Does Not Reveal Paragraph-Edit Agreement in Two Released Paper Acceptance Predictors

    Nov 4, 2026Shaoliang Yang, Jun WangAutomated EvaluationAutomated Peer Review

  13. GREP: Grounding Generated Room Evaluation in Principles

    Nov 4, 2026Mohan Cao, Yongsen Mao, Chucheng Xiang +4Automated EvaluationIndoor Scene Generation

  14. OUCH: Objective, Uniform, Controlled Haptic Stimulation for Automated Neurological Assessment

    Nov 3, 2026Sherry Choi, Sitao Sunny Li, Joseph Romero +4Automated EvaluationHaptic Feedback

  15. Therapy-FIS: A Dataset for Assessing Facilitative Interpersonal Skills in Psychotherapeutic Setting

    Nov 3, 2026Qijian Zheng, Wanqing Lin, Yuanfang Wang +4Automated EvaluationMental Health

  16. InformedXRD: Reproducible Benchmarks and Physics-Informed Evaluation for Powder Diffraction Symmetry Classification

    Nov 1, 2026Edward G Friedman, Elizabeth J. Baggett, Abhishek Shetty +5Benchmark ReproducibilityAutomated Evaluation

  17. CEFR Level Assessment across Languages and Text Granularities

    Oct 30, 2026Eugene Matsumura, Yuki AraseAutomated EvaluationMultilingual Text Classification

  18. A Novel Approach to Evaluating Evaluation Metrics for Multi-Output Structured Prediction

    Oct 29, 2026Akshay Vyas, Angelo Pimienta, Nicholas RuozziAutomated Evaluation

  19. Human Evaluation with Automatic Scores: Separating Sampling and Estimation Gains

    Oct 28, 2026Yang Liu, Ran Zou, Zhengtao Gui +1Human-in-the-Loop EvaluationLLM Evaluation

  20. Right Click, Wrong Box: Grading GUI Grounding by What a Click Does

    Oct 27, 2026Xiaokang Yuan, Binhui Yao, Luwei Yang +2GUI GroundingAutomated Evaluation

  21. MAREval: A Multi-Agent Framework for Evaluating Natural Language Recommendation Explanations

    Oct 26, 2026Reza Yousefi Maragheh, Jayesh Uddhav Kudase, Aysenur Inan +7Automated EvaluationLLM-as-a-Judge

  22. From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing

    Oct 26, 2026Harine Choi, Chenchen Kuai, Zihao Li +5LLM EvaluationLLM Auditing

  23. MechAlign: Evaluating Semantic Alignment at the Project Level in Mathematical Formalization

    Oct 26, 2026Hao Shen, Ruichen Qiu, Lihong Zhi +1Semantic AlignmentAutomated Evaluation

  24. Distilling Expert Judgment at Scale

    Oct 23, 2026Robbie Goldfarb, Andrew B. Hall, Jillian Fisher +2Automated EvaluationLLM-as-a-Judge

  25. MQAScore: Fine-Grained Text-Music Alignment Evaluation via Large Audio Language Model

    Oct 22, 2026Wei-Jaw Lee, Chun-Ping Wang, Xuanjun Chen +1Audio-Text AlignmentAutomated Evaluation

  26. VisCurate: Evaluating Behavioral Equivalence of Visual Skills

    Oct 21, 2026Md Farhan Ishraq, Md Farhan Ishmam, Reaz Hassan Joarder +1Automated Evaluation