Language Model Generation Evaluation

Overview

The field studies how to tell whether generated text is correct, useful, faithful to its sources, well written, and appropriate for a task. Approaches range from lexical and embedding-based metrics to human ratings, rubric-based LLM judges, learned evaluators, and task-specific checks such as code execution or schema validation. Benchmarks increasingly test long-form and open-ended outputs, multiple samples, languages, and settings, while evaluation-cost methods aim to select informative examples efficiently. A central challenge is that scores can depend on prompts, judges, and task context: a fluent or highly rated answer may still omit facts, repeat familiar ideas, or fail to reflect human preferences.

Latest papers

CardsList

422 papers32 papers in the last four weeks, down 50% on the four weeks before. 0.2% of all new papers.

  1. Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs

    Nov 5, 2024Mohammad Jalali, Azim Ospanov, Amin Gohari +1Generative AI EvaluationLanguage Model Generation Evaluation