Overview

The field asks whether generated text, images, audio, video, and other artifacts meet users’ goals—not just whether they resemble a reference. Common approaches include reference and embedding metrics, distribution distances such as FID and MMD, rubric-based LLM judges, pairwise preference comparisons, and expert or user studies. Task-specific benchmarks increasingly test dimensions such as faithfulness, constraint following, multimodal synchronization, robustness, and performance in complete workflows. A central challenge is that automated scores can disagree with people or miss important failures, so evaluations need clearer constructs, uncertainty estimates, human validation, and evidence of real-world utility.

Latest papers

CardsList

335 papers22 papers in the last four weeks, down 41% on the four weeks before. 0.2% of all new papers.

  1. Competition and Diversity in Generative AI

    Dec 11, 2024Manish RaghavanOutput Diversity in Language ModelsGenerative AI Evaluation

  2. Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs

    Nov 5, 2024Mohammad Jalali, Azim Ospanov, Amin Gohari +1Generative AI EvaluationLanguage Model Generation Evaluation

  3. Generating Interesting Scientific Ideas using Knowledge Graphs and LLMs: Evaluations with 100 Research Group Leaders

    May 27, 2024Xuemei Gu, Mario KrennAI-Assisted Scientific ResearchScientific Hypothesis Generation

  4. PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation

    Feb 6, 2024Pablo Lemos, Sammy Sharief, Esmeralda S. Whitammer +4Two-Sample TestingGenerative AI Evaluation