Overview

The field asks whether generated text, images, audio, video, and other artifacts meet users’ goals—not just whether they resemble a reference. Common approaches include reference and embedding metrics, distribution distances such as FID and MMD, rubric-based LLM judges, pairwise preference comparisons, and expert or user studies. Task-specific benchmarks increasingly test dimensions such as faithfulness, constraint following, multimodal synchronization, robustness, and performance in complete workflows. A central challenge is that automated scores can disagree with people or miss important failures, so evaluations need clearer constructs, uncertainty estimates, human validation, and evidence of real-world utility.

Latest papers

CardsList

335 papers22 papers in the last four weeks, down 41% on the four weeks before. 0.2% of all new papers.

  1. DrawScore: Evaluating Generated CAD Models against Engineering Drawings

    Nov 18, 2026Haorui Chen, Shiteng Lu, Zhanyu Guo +6Generative AI Evaluation3D Shape Matching

  2. VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications

    Nov 16, 2026Shuyi Ouyang, Xiaojun Tang, Zijian Zou +4Generative AI EvaluationAutomated Agent Evaluation

  3. Share your Screen and Talk to Gemini: How Young Learners Interact with Multimodal GenAI While Solving Scratch Tasks

    Nov 16, 2026Sven Jacobs, Lukas Berning, Natalie KieslerHuman-AI InteractionEducational Dialogue Systems

  4. Between Rigor and Reality: How AI Safety Benchmark Developers Understand Benchmark Use and Maintenance

    Nov 13, 2026Megan Li, Ningjing Tang, Christopher Stewart +2AI Safety EvaluationAI Safety

  5. OCTAVE: A Capability-Resolved Benchmark for End-to-End HTML5 Game Generation

    Nov 13, 2026Kehang Li, Xinlan Xu, Yulin Jin +10game generationCode Generation Benchmarks

  6. CADMA: Capacity-Aware Coverage Decomposition for Generative Model Assessment

    Nov 9, 2026Debin Meng, Zheng Gao, Ziquan Liu +3Generative AI EvaluationMode Collapse

  7. Discrete Diffusion Playground: A 2D Benchmark for Discrete Generative Models

    Nov 8, 2026Justin Shao, Sihyun Park, David Ryan Koes +1discrete generative modelingGenerative Modeling

  8. MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification

    Nov 7, 2026Jiaju Ma, Ashvini Kumar Jindal, Chuhua Wang +1Generative AI Evaluation

  9. The Invisible Gap: Romanized Indic Generation Fails Where Automatic Metrics Succeed

    Nov 6, 2026Madhu Sudhan Rao Beesetty, Surbhi Sharma, Devesh Maheshwari +1Language Model Generation EvaluationGenerative AI Evaluation

  10. Contextuality and Contradictions: Exploring Real-World Identifications of AI-Generated Writing

    Nov 5, 2026Michelle Morgenstern, Julia Witte Zimmerman, Tabia Tanzin Prama +5AI-Generated Text DetectionGenerative AI Evaluation

  11. PATTS: Parallel-Aware Test-Time Scaling for Thought-Image Generation in Multimodal Discrete Diffusion LLMs

    Nov 3, 2026Yasong Dai, Zeeshan Hayder, David Ahmedt-Aristizabal +1Test-Time Scaling for VLMsDiffusion VLMs

  12. Benchmark Prediction with Sparse and Unpaired Cached Responses

    Nov 2, 2026Hayden Helm, Avijit Ghosh, Ben Johnson +1Generative AI EvaluationPredictive Representation Learning

  13. Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation

    Oct 31, 2026Gustavo Tironi, Felipe Maia Polo, Diego MesquitaGenerative AI EvaluationUncertainty Quantification

  14. Generative AI in Turbulence: Foundations, Applications, Validations and Benchmarks

    Oct 26, 2026Luca Biferale, Michele BuzzicottiGenerative ModelingGenerative AI Evaluation

  15. A Battery for AI Propensity Measurement: Rubrics, Synthetic Item Banks, and Annotations

    Oct 25, 2026Peter Romero, Haotian Li, Hugo Save +8Generative AI EvaluationSynthetic Benchmark Generation

  16. Finite-Sample Sensitivity of Kernel Metrics for Generative Evaluation

    Oct 25, 2026Tomasz Radzikowski, Mateusz ModrzejewskiGenerative AI EvaluationKernel Methods

  17. How Geometry Shapes Generative Model Evaluation on Riemannian Manifolds

    Oct 24, 2026Misha Oberski, Valentijn Oldenburg, Xanthos Kekkos +6Generative AI EvaluationWasserstein Distance

  18. NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting

    Oct 23, 2026Matt Wilde, Jillian Fisher, Andrew B. Hall +2Generative AI EvaluationFactual Consistency Evaluation

  19. TESSA: A Governance-Oriented Custom GPT for SME Guidance under the EU AI Act: Design, Implementation, and Evaluation

    Oct 22, 2026Chalisa Veesommai Sillberga, Kai-Kristian Kemella, Pekka Sillberga +3AI GovernanceGenerative AI Evaluation

  20. Wet-Biased Data Pipelines Mislead the Training and Evaluation of Generative Precipitation Nowcasting

    Oct 19, 2026Christoph Metzl, Stefan Kapsreiter, Kianusch Vahid Yousefnia +4Precipitation NowcastingGenerative AI Evaluation

  21. Can AI interview? Comparing AI-led to Human-led Chat Interviews

    Oct 16, 2026Semra Guven, Tobias Gårdhus, Andreas Bjerre-Nielsen +2Human-AI InteractionGenerative AI Evaluation

  22. Watts If? A Benchmark for Counterfactual Reasoning About the AI Data Center Buildout

    Oct 15, 2026Pankaj Chouhan, Rajarshi Das, Aditya Dubey +5Counterfactual Reasoning in Language ModelsGenerative AI Evaluation

  23. Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans

    Oct 14, 2026Chayapatr Archiwaranguprok, Krystal Kirkland, Rachel Poonsiriwong +17Generative AI EvaluationLLM Evaluation