Recent advances in language models have prompted a growing body of work that questions prevailing evaluation practices. However, many such critiques have already been discussed in natural language processing (NLP), a field with a long history of methodological reflection on evaluation. We conduct a scoping review spanning four decades and develop a taxonomy of 22 recurring concerns. For each concern, we examine how recently it was discussed in the reviewed literature and whether it receives explicit attention in institutional checklists, finding that some concerns have become less visible in literature and many lack coverage in these checklists. Our work provides an overview of the major concerns in evaluation methodology that have shaped the field and identifies potential targets for renewed investigation and field-level intervention by the research community.
Figures & tables
Figure 1: Temporal distribution of survey papers addressing evaluation methodology (1981–2024) across the four main concern types.
Dimension
ACL
Broader ML
( n=192 )
( n=65 )
Data
48 (25.0%)
29 (44.6%)
Metrics
90 (46.9%)
12 (18.5%)
Hypothesis
42 (21.9%)
27 (41.5%)
Reporting
17 (8.9%)
7 (10.8%)
Table 1: Taxonomy dimensions represented in the ACL and broader ML collections, with entries showing paper counts and percentages within each collection.
Figure 2: Evaluation concerns across research areas showing the percentage of papers within each area assigned to each taxonomy dimension.
Figure 3: A taxonomy of evaluation concerns in NLP across four dimensions: Data , Metrics , Hypothesis , and Reporting . Terminal nodes represent the 22 annotated concerns; intermediate nodes group related concerns.
Figure 4: Temporal coverage of the 22 evaluation concerns in the 257-paper corpus. Cells show tagged-paper counts; blank cells indicate zero papers. Label colours follow the taxonomy, and period totals appear below the horizontal axis.
Figure 5: Changes in the share of papers addressing each concern. Light-blue circles represent 1981–2021 ( n=177 ); dark-blue diamonds represent 2022–2024 ( n=80 ). Panels show the 11 concerns with increasing share (a) and the 11 with decreasing share (b), using the same scale.
Evaluation of large language models (LLMs) remains largely pre-scientific: it involves many design decisions, with little guidance on how to make them. These choices can substantially affect measured performance and, consequently, decisions about subsequent pretraining. To identify effective evaluation setups, we investigate 8 aspects, such as task formulation or the number of in-context examples. We compare the setups using three widely-used signal criteria, repurposed as normative meta-measures: monotonicity, non-randomness, and ranking consistency of the evaluation scores across intermediate checkpoints. In an early pretraining scenario, we evaluate these setups on 26 monolingual 2B-parameter models trained on different types of data, using 80 tasks in four languages. We also examine the setups on three larger LLMs (≥7B) in the same four languages and explore how the preferred setups change with model scale. Our results yield concrete recommendations: some choices, such as the use of multiple prompts, are consistently useful, whereas others, such as task formulation, depend on model scale. More generally, our framework serves as a recipe for the choice of evaluation setups in new scenarios.
Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task.
Benchmarks are essential for evaluating and improving large language models (LLMs). Yet many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Benchmark compression lowers this cost by estimating full-benchmark scores and rankings from a partial evaluation. Research in this area has grown rapidly since 2024, but to our knowledge no survey has systematically organized its methods and findings. We review 119 papers to explain how methods choose what to evaluate, recover full-benchmark results, and verify their reliability. From this analysis, we develop a taxonomy covering compression within individual benchmarks, across benchmark suites, and during adaptive evaluation, together with work on redundancy and reliability. We compare their information needs, total costs, reliability checks, and limitations. Methods reduce evaluation costs by selecting representative subsets, using cheaper or reusable information, and testing adaptively, but risk poor coverage, inaccurate estimates, and sequential delays. The resulting taxonomy and practical guide help researchers choose, validate, and deploy methods for more efficient and reliable LLM evaluation.