Generative AI Evaluation
Overview
The field asks whether generated text, images, audio, video, and other artifacts meet users’ goals—not just whether they resemble a reference. Common approaches include reference and embedding metrics, distribution distances such as FID and MMD, rubric-based LLM judges, pairwise preference comparisons, and expert or user studies. Task-specific benchmarks increasingly test dimensions such as faithfulness, constraint following, multimodal synchronization, robustness, and performance in complete workflows. A central challenge is that automated scores can disagree with people or miss important failures, so evaluations need clearer constructs, uncertainty estimates, human validation, and evidence of real-world utility.
Latest papers
313 papers38 papers in the last four weeks, up 3% on the four weeks before. 0.2% of all new papers.