Automated Evaluation

Automated evaluation studies how to measure the quality, correctness, safety, and usefulness of AI outputs and systems without relying entirely on manual review. Common approaches include reference-based metrics, reference-free scoring, LLM-as-a-judge, rubric and checklist scoring, pairwise comparisons, and executable tests that verify code, tools, or interactive behavior. Benchmarks increasingly assess agents through full task trajectories and evidence from browsers, databases, or other environments, while statistical calibration and targeted human review help interpret results. The central challenge is making automated scores valid, robust across tasks and populations, and resistant to bias or gaming; agreement with human ratings alone does not guarantee that an evaluator measures the intended capability.