EvalCards: A Framework for Standardized Evaluation Reporting
Organizations: University of Copenhagen · ETH Zurich · University of Amsterdam · University of Barcelona ·
Abstract
Evaluation reports guide the use of general-purpose AI (GPAI) models, yet inconsistent reporting often leaves essential questions unanswered. Through a review of prior work, we identify three central issues in evaluation reporting practices today, reproducibility, accessibility, and accountability, and examine how existing reporting artifacts fall short in addressing them. We introduce the EvalCards framework, outlining its key dimensions and presenting a preliminary format for concise evaluation summaries. We conclude by discussing how the framework can guide the future development of evaluation reporting standards.
Figures & tables
| Issue | Definition | Representative works |
|---|---|---|
| Reproducibility | The gaps in methodological reporting that limit stakeholders’ ability to reproduce or independently verify evaluation results. | Belz et al. (2021) , Palen-Michel et al. (2021) , Storks et al. (2023) , Belz et al. (2023b) , Laskar et al. (2024a) , Biderman et al. (2024) |
| Accessibility | The barriers to finding and interpreting evaluation results that limit stakeholders’ ability to make informed comparisons and decisions. | Ethayarajh and Jurafsky (2020) , Crisan et al. (2022) , Liao et al. (2023) , Burnell et al. (2023) , Sokol et al. (2024) , Zhao et al. (2025c) |
| Accountability | The gaps in evaluation evidence and mechanisms for challenge and response that limit stakeholders’ ability to assess model risks and exercise oversight. | Raji et al. (2020) , Wieringa (2020) , Cattell et al. (2024) , Weidinger et al. (2024) , McGregor et al. (2025) , Longpre et al. (2025) |
Generative AI Evaluation
Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.
See all 311 papers