Evaluation reports guide the use of general-purpose AI (GPAI) models, yet inconsistent reporting often leaves essential questions unanswered. Through a review of prior work, we identify three central issues in evaluation reporting practices today, reproducibility, accessibility, and accountability, and examine how existing reporting artifacts fall short in addressing them. We introduce the EvalCards framework, outlining its key dimensions and presenting a preliminary format for concise evaluation summaries. We conclude by discussing how the framework can guide the future development of evaluation reporting standards.
Figures & tables
Figure 1: Tagged corpus with three-way reporting issue classification based on Table 1 , grouped by publication year. Each paper supports multiple tags.
Issue
Definition
Representative works
Reproducibility
The gaps in methodological reporting that limit stakeholders’ ability to reproduce or independently verify evaluation results.
Belz et al. (2021) , Palen-Michel et al. (2021) , Storks et al. (2023) , Belz et al. (2023b) , Laskar et al. (2024a) , Biderman et al. (2024)
Accessibility
The barriers to finding and interpreting evaluation results that limit stakeholders’ ability to make informed comparisons and decisions.
Ethayarajh and Jurafsky (2020) , Crisan et al. (2022) , Liao et al. (2023) , Burnell et al. (2023) , Sokol et al. (2024) , Zhao et al. (2025c)
Accountability
The gaps in evaluation evidence and mechanisms for challenge and response that limit stakeholders’ ability to assess model risks and exercise oversight.
Raji et al. (2020) , Wieringa (2020) , Cattell et al. (2024) , Weidinger et al. (2024) , McGregor et al. (2025) , Longpre et al. (2025)
Table 1: Three umbrella issues in evaluation reporting identified through critical synthesis.
Figure 2: Issues in evaluation reporting and possible mitigation pathways for each, offering design directions for future reporting artifacts.
Figure 3: The EvalCards framework and the key considerations within its three dimensions.
AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs. The cost is interpretive: readers cannot reliably compare results across sources, identify what a report omits, or trace an aggregate claim to its underlying evidence. Recent efforts address isolated components but leave three gaps: they cover only narrow slices of the evaluation lifecycle and do not compose into a single interpretable record; they specify static representations that do not differentiate the questions different stakeholders bring to the same evidence; and they remain proposals on paper, lacking the extraction infrastructure required for adoption at scale. We present \EvalCards{}, an operational reporting layer that composes benchmark metadata, evaluation run data, and model metadata into a unified record. We (1) derive a reporting schema from a structured review of 52 papers and 10 stakeholder interviews, (2) implement four interpretive signals (reproducibility, documentation completeness, provenance and risk, and score comparability), rendered through reader modes calibrated to research and non-research audiences, and (3) deploy a monitoring tool that applies \EvalCards{} across 5,816 models, 635 benchmarks, and 101,843 results, surfacing systematic gaps in current reporting practice.
The evaluation of explainable AI (XAI) methods is affected by a lack of standardization. Metrics are inconsistently defined, incompletely reported, and rarely validated against common baselines. In this paper, we identify transparency of evaluation reporting as a central, under-addressed problem. We propose the XAI Evaluation Card, a documentation template analogous to model cards, designed to accompany any study that introduces an XAI evaluation metric. The card covers explicit declaration of target properties, grounding levels, metric assumptions, validation evidence, gaming risks, and known failure cases. We argue that adopting this template as a community norm would reduce evaluation fragmentation, support meta-analysis, and improve accountability in XAI research.
AI evaluations are widely used for testing and understanding progress. However, the diverse evaluators bring with them inconsistencies that challenge analysis and comparison. First, results are saved in incompatible formats, scattered across leaderboards, papers, blog posts, evaluation harness logs, and custom repositories. Second, results are created by different evaluation frameworks, which produce divergent scores for nominally identical evaluations and record metadata inconsistently, hindering comparison, cross-community evaluation science, cost reduction, and reuse. We introduce Every Eval Ever, the first shared schema and community-crowdsourced repository for AI evaluation results. The schema standardizes how evaluations are represented in a unified, single JSON document. It is source-agnostic by design, ingesting results from evaluation harnesses and papers alike, and optionally stores per-instance outputs for fine-grained analysis. We contribute: (i) a community-governed metadata schema with a companion instance-level schema, the first standardization effort of its kind; (ii) automatic converters from popular formats, evaluation harnesses, and leaderboards to the unified schema; and (iii) a crowdsourced community database hosted on Hugging Face, currently spanning to date 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats.
Jan Batzner, Sree Harsha Nelaturu, Damian Stachura +45