Human-in-the-Loop Evaluation

This field examines how people evaluate AI outputs and collaborate with AI during testing and deployment, including whether systems are accurate, useful, safe, and appropriately trusted. Common approaches include expert rubrics and pairwise preferences, user studies and interaction traces, human-validated benchmarks, and LLM judges calibrated against human ratings, with uncertain or disputed cases escalated for review. Researchers also study how to allocate limited review effort, model annotator disagreement, and measure effects on human decisions and learning over time. The central tension is scaling evaluation with automated judges and simulations without losing expert context or mistaking agreement, confidence, or benchmark scores for real-world quality.