What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025
Authors: Jing Yang, Nils Feldhus, Salar Mohtaj, Leonhard Hennig, Qianli Wang, Eleni Metheniti, Sherzod Hakimov, Charlott Jakob, Veronika Solopova, Konrad Rieck, David Schlangen, Sebastian Möller, Vera Schmitt
Organizations: Technische Universität Berlin · BIFOLD – Berlin Institute for the Foundations of Learning and Data · German Research Center for Artificial Intelligence (DFKI), Berlin · ANITI · University of Potsdam · CERTAIN
As Natural Language Generation (NLG) dominates modern NLP, scalable evaluation remains a critical bottleneck. Consequently, LLM-as-a-judge (LaaJ) adoption has accelerated rapidly, appearing in more papers than human evaluation in 2025. This pivotal shift motivates a critical analysis of current evaluation practices. Overcoming the limits of rigid keyword filtering and manual review, we employ a multi-LLM information extraction pipeline to gather structured metadata from 14,171 papers across four major NLP conferences (2020-2025). Analyzing 3,334 filtered NLG papers, we identify three systemic challenges. (1) Metric inertia: despite the shift toward open-ended generation, legacy lexical metrics (BLEU, ROUGE) persist as primary indicators, typically used alongside rather than replaced by semantic alternatives. (2) Metric-criteria mapping problem: our paper-level co-occurrence data reveals that general-purpose automatic metrics are applied as broad proxies for quality, without specifying which dimension of text generation they are intended to evaluate. (3) Validation gap: LaaJ has grown rapidly without commensurate human validation (fewer than 8% of papers). Crucially, while LaaJ correlates with aggregate quality, alignment collapses on fine-grained criteria like fluency. To address these gaps, we distill our findings into a minimal Evaluation Checklist to guide metric selection, construct validity, and LaaJ deployment.
Figures & tables
Figure 1: Evaluation method adoption across the 3,334 Top-30 NLG-task papers (2020–2025). Automatic metrics remain ubiquitous ( ∼ 95%). LaaJ adoption surged from less than 1% before 2023 to 46% in 2025, surpassing human evaluation, which declined from 51% to 43%. Yet only 16% of papers in 2025 explicitly validate LaaJ against human evaluation, indicating a validation gap.
Figure 2: Our paper annotation pipeline (§ 3 ), including converting PDF to text (§ 3.1 ), extraction of metadata based on four NLG evaluation questions (Table 15 ), harmonization based on majority vote, filtering, and merging (§ 3.2 ), and human validation (§ 4 ), yielding 14k structured papers including a subset of 110 papers with gold annotations.
Figure 3: Per-task evaluation method adoption (%) of top-four NLG tasks per year. The filled area shows that task’s share of all NLG papers each year.
Figure 4: Evaluation trends for the four most frequent NLG tasks (columns) under three evaluation paradigms (rows). Each panel is a separate bump chart: the y -axis gives the annual frequency rank (1 = most used) of the eight most common metrics for that task, from 2020 to 2025 on the x -axis. Marker size scales with the metric’s likelihood ratio (LR) for the task in that year. Solid lines with filled markers mark a significant overall association with the task; dashed lines with hollow markers mark metrics that are frequent but have a low LR. Endpoint labels give the metric name and (overall LR, k ) , where k is the number of papers. COMET’s (19.0, 151) in the MT/Automatic Metrics panel thus means it appears in 151 MT papers and is 19 times more likely to appear in MT than elsewhere; it enters at rank 4 in 2021 and holds rank 2 from 2022 onward, behind BLEU. Dotted arcs bridge years in which a metric drops out of the top eight; dashed connectors attach labels to metrics that are outside the top eight in 2025. A missing line in a given year means the metric was not used. Color denotes the QCET L1 dimension: QO (intrinsic to output), QI (relative to input), QT (relative to target reference), QE (relative to external standard), and AUX (QCET extensions).
Figure 5: Within-task co-occurrence LRs for metric–criterion pairs, one panel per task. Rows list metrics and columns list criteria; each axis contains the five most frequent items (black labels) and the five with the highest task-LR (red labels), so the quadrant where frequent metrics meet frequent criteria (yellow background) is distinct from the rest. Each cell is split along the diagonal: the upper-left orange triangle covers the task’s LaaJ papers and the lower-right blue triangle its human-evaluation papers. Colour intensity encodes the co-occurrence LR, and an outline marks a significant pair. In Dialogue Generation, BLEU with Fluency is outlined and shaded on both halves (133 human-evaluation papers, LR=2.02 ; 19 LaaJ papers, LR=2.59 ), indicating that the pairing is associated with the task under both paradigms, whereas a cell shaded on one side only reflects a paradigm-specific pairing. Across tasks, papers pair metrics and criteria along frequently used terms rather than task-distinctive ones.
Figure 6: Distribution of LaaJ vs. human evaluation scores by metric and criterion. Individual observations are shown as gray dots. n denotes the number of studies included. Values are based on human annotated 70 papers.
Variable (reference)
LR [95% CI]
LaaJ use
1.28 [1.16, 1.41]
Year 2021 (vs. 2020)
1.11 [0.98, 1.27]
Year 2022
1.08 [0.95, 1.23]
Year 2023
1.12 [0.99, 1.27]
Year 2024
0.97 [0.84, 1.11]
Year 2025
0.86 [0.74, 1.00]
Table 2: Regression result. Bold: 95% CI excludes 1, significant. Year and venue are exclusive categories, compared to a reference (2020 / ACL); task indicators are not exclusive (a paper can address several tasks), so each task LR compares papers with vs. without that task
Table 3: The 6 extension leaves added to QCET (111 → 117). Origin indicates whether the leaf was discovered by HDBSCAN clustering of stage-1 classification residuals ( cluster ) or by frequency analysis of remaining unclustered residual variants ( frequency ). Variants is the number of distinct raw criterion strings mapped to this leaf; Paper-occ. sums the LaaJ and human paper-occurrences across those variants.
Table 7: Top-10 human evaluation criteria after QCET-based normalization, with paper count and the most frequent raw variants that mapped to each target.
Term
Cnt
Top Variants (count)
Chinese
1142
Chinese (1106); Mandarin (18); Simplified Chinese (6)
German
1034
German (1031); Swiss German (1); Swiss-German (1)
Spanish
537
Spanish (536); spa (1)
Russian
402
Russian (401); rus (1)
Arabic
322
Arabic (288); Egyptian Arabic (13); Modern Standard Arabic (7)
Portuguese
219
Portuguese (210); Brazilian Portuguese (9)
Appendix
Table 8: Top-10 normalized languages and their top variants
Term
Cnt
Top Variants (count)
GSM8K
454
GSM8K (428); GSM8k (18); GSM-8K (6)
CNN/DailyMail
303
CNN/DailyMail (159); CNN/Daily Mail (70); CNN/DM (51)
Table 10: Top-10 normalized NLG models and their top variants
Excluded task
Reason
Code Generation
Not natural language output; verified by execution, not text quality
Mathematical Reasoning
Evaluated by answer correctness against gold answer, not linguistic criteria
Math Problem Solving
Same as Mathematical Reasoning
Math Reasoning
Same as Mathematical Reasoning
Text-To-SQL Generation
Query output; verified by execution accuracy
Semantic Parsing
Meaning representation; evaluated by exact/graph match
Appendix
Table 11: Excluded Tasks and Reasons
Top excluded tasks
Top-3 evaluation metrics
Code Generation (401)
Accuracy (33%), pass@k (30%), Exact Match (15%)
Instruction Following (131)
Accuracy (41%), Win Rate (28%), ROUGE (16%)
Math Reasoning, all variants (154)
Accuracy (73%), pass@k (24%), Exact Match (14%)
Text-to-SQL Generation (41)
Execution Accuracy (46%), Exact Match Accuracy (32%), Exact Match (24%)
Semantic Parsing (55)
Exact Match (36%), Accuracy (29%), Exact Match Accuracy (24%)
Language Modeling (102)
Perplexity (78%), Accuracy (36%), BLEU (24%)
Appendix
Table 12: Top-3 metrics for newly added tasks
Field
Mean (1–3)
Q1: tasks
2.62
Q1: datasets
2.71
Q1: languages
2.88
Q1: NLG models
2.52
Q1: output formats
2.52
Q2: automatic metrics
2.69
Appendix
Table 13: LLM-extraction field-level Likert scores (60 papers; 1=mismatch, 2=partial, 3=full match). Means above 2.5 across all fields indicate that LLM extraction substantially aligns with human annotators.
Example A
Example B
(QA → exact match)
(coverage → rel. factual accuracy)
N (papers)
3,334
1,711
k11 ( A and B )
136
3
k12 ( A , not B )
326
19
k21 ( B , not A )
90
25
k22 (neither)
2,782
1,664
Appendix
Table 14: LR and G2 on two real pairs from our data with nearly the same LR but opposite significance outcomes. Example A (task–metric): A = question answering, B = exact match; full set ( N=3,334 ). Example B (metric–criterion): A = coverage, B = relative factual accuracy (QCET QEC-c-2); human-evaluation subset ( N=1,711 ). The large difference in k11 (136 vs. 3) drives the difference in E11 and hence G2 , producing opposite conclusions after BH-FDR correction.
Key
Description
Q1: NLG task presence ( answer_1 )
answer
“Yes” if the paper addresses any NLG task.
quote
Verbatim excerpt supporting the decision.
tasks
NLG tasks (e.g., Summarization, MT, Paraphrase Gen, or Other:<task> ).
datasets
List of datasets used for generation/evaluation.
languages
List of languages (e.g., English, Chinese).
Appendix
Table 15: Schema of the JSON object returned by the LLM for each paper.
Figure 7: Heatmap of task-year distribution of the top-30 tasks. Both the total number and percentage per year are shown. The percentage of MT is decreasing while QA-related tasks increase significantly (QA, VQA).
Figure 8: Distribution of metadata by tasks, columns include top-four and all-tasks, and rows are the metadata extracted (counts are after term normalization).
Task
Criterion
Recommended Metric(s)
Avoid as Primary
LaaJ viable?
Key References
Machine Translation
Adequacy / Translation Quality
COMET Rei et al. (2020) , MetricX-25 Juraska et al. (2025) (COMET 39.0% of 2025 MT; BLEU still 61.8%)
BLEU (task-LR =2.63 , low criterion-LR)
✓(system-level; segment-level weaker Lavie et al. (2025) )
Lavie et al. (2025) ; Mathur et al. (2020)
Dialogue Generation
Diversity
Distinct- n , Self-BLEU Li et al. (2016)
BLEU (task-LR ≈1 for DG)
✗(auto metrics sufficient)
Li et al. (2016)
Engagement / Appropriateness
LaaJ + human (multi-item Likert)
ROUGE (n-gram overlap)
✓(rubric)
van der Lee et al. (2019) ; Bavaresco et al. (2025)
Safety / Harmlessness
LaaJ rubric + human red-teaming (47 DG papers use LaaJ with safety criteria)
—
✓(validate vs. humans; judge: GPT-4/GPT-4o)
Bavaresco et al. (2025) ; Wang et al. (2024)
Text Summarization
Consistency / Faithfulness (vs. input)
SummaC Laban et al. (2022) (28), UniEval Zhong et al. (2022) (23)
ROUGE (low criterion-LR with consistency)
✓(validation evidence thin, § 5.3 )
Laban et al. (2022) ; Zhong et al. (2022)
Factual Accuracy
QAFactEval Fabbri et al. (2022) (12), MiniCheck Tang et al. (2024) (7, mostly 2024–2025)
ROUGE / BERTScore (similarity = factuality)
✓(validation required, § 5.3 )
Fabbri et al. (2022) ; Tang et al. (2024)
Appendix
Table 19: Task-specific evaluation guidance derived from our empirical analysis (§ 5 ). “Avoid as Primary” lists metrics that co-occur frequently but exhibit low LR with the target criterion, indicating inertial use rather than validated selection. ✓ and ✗ indicate whether an LLM-as-a-judge evaluator is recommended as a viable complement. All recommendations should be validated on the specific task and language before deployment.
Natural Language Generation (NLG) evaluation has changed dramatically since 1990, and will continue to evolve in the future. In 1990, when NLG had close ties to linguistics, there was very little formal experimental evaluation in the modern sense. In 2026, when NLG is closely linked to machine learning, experimental evaluation is expected and indeed fundamental to research. Many evaluation techniques were developed over this period, including most recently LLM-as-Judge. I expect NLG evaluation will continue to evolve in the future. In particular, impact, qualitative, and safety evaluation will become more important as large numbers of people routinely use NLG technology.
Ehud Reiter
Dept of Computing Science University of Aberdeen Aberdeen, UK
Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge.
Lukáš Eigler, Jindřich Libovický, David Hurych
Faculty of Mathematics and Physics, Charles University, Czech Republic
The LLM-as-a-judge (LLMaJ) framework has emerged as a promising solution for cheap, reproducible, reference-free Natural Language Generation (NLG) evaluation. Prior work seeks to improve LLMaJ by decomposing evaluation tasks into simpler sub-tasks. In this work, we systematically compare LLMaJ methods with and without decomposition on multiple NLG datasets. We find no evidence that LLMaJ with task decomposition leads to performance gains over a fair baseline that does not use decomposition. Instead, we find that previously reported performance gains in decomposition-based LLMaJ stem from using human labels as training data, and not task decomposition itself. Also, we find that, when human labels are available, LLMaJ without using task decomposition can perform comparably to human annotators.
Sebastian Steindl, Nikos Voskarides, Alberto Gasparin +1