Overview

Automated evaluation studies how to measure the quality, correctness, safety, and usefulness of AI outputs and systems without relying entirely on manual review. Common approaches include reference-based metrics, reference-free scoring, LLM-as-a-judge, rubric and checklist scoring, pairwise comparisons, and executable tests that verify code, tools, or interactive behavior. Benchmarks increasingly assess agents through full task trajectories and evidence from browsers, databases, or other environments, while statistical calibration and targeted human review help interpret results. The central challenge is making automated scores valid, robust across tasks and populations, and resistant to bias or gaming; agreement with human ratings alone does not guarantee that an evaluator measures the intended capability.

Latest papers

CardsList

422 papers28 papers in the last four weeks, down 35% on the four weeks before. 0.2% of all new papers.

  1. A Course Intelligence Platform for Higher Education: Lessons from AI-Assisted Course Evaluation

    Nov 3, 2024Bo Yuan, Jiazi Hu, Haimei ZhaoEducational AssessmentGenerative AI in Education

  2. GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models

    Dec 11, 2023Jiaxu Zhao, Meng Fang, Shirui Pan +2Language Model Bias EvaluationAutomated Evaluation