Generative AI Evaluation

Follow this topic Ask about this topic

The field asks whether generated text, images, audio, video, and other artifacts meet users’ goals—not just whether they resemble a reference. Common approaches include reference and embedding metrics, distribution distances such as FID and MMD, rubric-based LLM judges, pairwise preference comparisons, and expert or user studies. Task-specific benchmarks increasingly test dimensions such as faithfulness, constraint following, multimodal synchronization, robustness, and performance in complete workflows. A central challenge is that automated scores can disagree with people or miss important failures, so evaluations need clearer constructs, uncertainty estimates, human validation, and evidence of real-world utility.

Latest papers

313 papers38 papers in the last four weeks, up 3% on the four weeks before. 0.2% of all new papers.

AtomBench: A Benchmarking Framework for Generative Crystal Reconstruction Models in Conventional Superconductors
Oct 17, 2025Charles Rhys Campbell, Aldo H. Romero, Kamal ChoudharyMaterials Science Generative Modeling
Department of Physics and Astronomy, West Virginia University, Morgantown, WV 26506, USA · Department of Materials Science and Engineering, Whiting School of Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA · Department of Electrical and Computer Engineering, Whiting School of Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA
Generative AI and Sales Productivity: Field Experiments in Online Retail
Oct 14, 2025Lu Fang, Zhe Yuan, Kaifu Zhang +2Generative AI Evaluation
†School of Management, Zhejiang University of Finance & Economics; Laboratory for Human-AI Collaboration and Digital Intelligence Innovation, Zhejiang University · ‡Corresponding author. Center for Research of Private Economy, School of Economics, Future Regional Development Laboratory, and Academy of Financial Research, Zhejiang University · §Partner Company +2
Attribution Gradients: Incrementally Unfolding Citations for Critical Examination of Attributed AI Answers
Oct 1, 2025Hita Kambhamettu, Alyssa Hwang, Philippe Laban +1Information Design Claim Attribution in Language Models
University of Pennsylvania, Philadelphia, PA, USA · Microsoft Research, New York, NY, USA
Measuring AI "Slop" in Text
Sep 23, 2025Chantal Shaib, Tuhin Chakrabarty, Diego Garcia-Olano +1Generative AI Evaluation Language Model Generation Evaluation
Northeastern University · Stony Brook University · Meta AI
Generative AI performance in core undergraduate mathematics: a curriculum-level case study
Sep 15, 2025Benjamin J. Walker, Nikoleta Kalaydzhieva, Beatriz Navarro Lameda +1Educational Assessment Generative AI in Education
Department of Mathematics, University College London, Gordon Street, WC1H 0AY, London, UK
Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation
Aug 19, 2025Krishna Garg, Firoz Shaik, Sambaran Bandyopadhyay +1LLM Evaluation Language Model Generation Evaluation
University of Illinois Chicago · Adobe Research
Evaluating Style-Personalized Text Generation: Challenges and Directions
Aug 8, 2025Anubhav Jangra, Bahareh Sarrafzadeh, Silviu Cucerzan +2LLM-as-a-Judge Language Model Generation Evaluation
Columbia University, USA · Microsoft, USA · The University of York, UK
PALATE: Peculiar Application of the Law of Total Expectation to Enhance the Evaluation of Deep Generative Models
Mar 24, 2025Tadeusz Dziarmaga, Marcin Kądziołka, Laura Musioł +4Generative AI Evaluation Generative Modeling
Jagiellonian University, Faculty of Mathematics and Computer Science, Kraków, Poland · Jagiellonian University, Doctoral School of Exact and Natural Sciences, Kraków, Poland
Competition and Diversity in Generative AI
Dec 11, 2024Manish RaghavanOutput Diversity in Language Models Generative AI Evaluation
MIT Sloan School of Management & Department of Electrical Engineering and Computer Science.
Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs
Nov 5, 2024Mohammad Jalali, Azim Ospanov, Amin Gohari +1Generative AI Evaluation Language Model Generation Evaluation
Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Information Engineering, The Chinese University of Hong Kong
Generating Interesting Scientific Ideas using Knowledge Graphs and LLMs: Evaluations with 100 Research Group Leaders
May 27, 2024Xuemei Gu, Mario KrennAI-Assisted Scientific Research Scientific Hypothesis Generation
Max Planck Institute for the Science of Light, Staudtstrasse 2, 91058 Erlangen, Germany
PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation
Feb 6, 2024Pablo Lemos, Sammy Sharief, Esmeralda S. Whitammer +4Two-Sample Testing Generative AI Evaluation
1Mila – Québec Artificial Intelligence Institute · 2Ciela Institute · 3Université de Montréal +3
Consistent Distributed Ranking of Generative Models via Kernel Distances
Oct 18, 2023Zixiao Wang, Farzan Farnia, Zhenghao Lin +2Kernel Methods Image Generation Evaluation

Oct 17, 2025J/K move · Enter open · S save

AtomBench: A Benchmarking Framework for Generative Crystal Reconstruction Models in Conventional Superconductors

Charles Rhys Campbell, Aldo H. Romero, Kamal Choudhary

Department of Physics and Astronomy, West Virginia University, Morgantown, WV 26506, USA · Department of Materials Science and Engineering, Whiting School of Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA · Department of Electrical and Computer Engineering, Whiting School of Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA

Materials Science Generative Modeling Generative AI Evaluation

Open paper AskPDF arXiv

A key question in benchmarking generative crystal reconstruction models is how the amount and type of crystallographic information provided to a generative model affects its ability to reconstruct atomic structures. Yet such comparisons often overlook the fact that models receive unequal information about the target during reconstruction, thereby confounding architectural conclusions. We present AtomBench, an extensible, model-agnostic framework for comparing generative models on a well-defined crystal reconstruction task (rather than de novo generation), which we here apply to conventional superconductors. We train and evaluate four models, AtomGPT, CDVAE, FlowMM, and MatterGen, on the JARVIS Supercon-3D and Alexandria DS-A/B datasets, grouping them by the information each accesses at inference. Reconstruction fidelity is measured by the Kullback-Leibler divergence (KLD) and mean absolute error (MAE) of lattice parameters and the root-mean-squared displacement (RMSD) of atomic coordinates. We further introduce the continuous corrected RMSD (ccRMSD), a continuous measure of local geometric fidelity defined for every structure in the test set. MatterGen achieves the best atomic-coordinate reconstruction, followed by AtomGPT, while CDVAE reconstructs lattices most accurately, and FlowMM is the least accurate but fastest overall. We find that conditioning on critical temperature T

_c

does not consistently improve fidelity. We also release AtomBench as an open-source Python package that reproduces all reported reconstruction metrics, figures, and tables from one or more benchmark files and supports direct submission to the JARVIS-Leaderboard. Any inverse model emitting crystal reconstructions can be benchmarked with atombench, and we encourage community use. https://github.com/atomgptlab/atombench