cs.CLJan 12, 2026
SaveWhat Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025
Organizations: Technische Universität Berlin · BIFOLD – Berlin Institute for the Foundations of Learning and Data · German Research Center for Artificial Intelligence (DFKI), Berlin · ANITI ·
Abstract
As Natural Language Generation (NLG) dominates modern NLP, scalable evaluation remains a critical bottleneck. Consequently, LLM-as-a-judge (LaaJ) adoption has accelerated rapidly, appearing in more papers than human evaluation in 2025. This pivotal shift motivates a critical analysis of current evaluation practices. Overcoming the limits of rigid keyword filtering and manual review, we employ a multi-LLM information extraction pipeline to gather structured metadata from 14,171 papers across four major NLP conferences (2020-2025). Analyzing 3,334 filtered NLG papers, we identify three systemic challenges. (1) Metric inertia: despite the shift toward open-ended generation, legacy lexical metrics (BLEU, ROUGE) persist as primary indicators, typically used alongside rather than replaced by semantic alternatives. (2) Metric-criteria mapping problem: our paper-level co-occurrence data reveals that general-purpose automatic metrics are applied as broad proxies for quality, without specifying which dimension of text generation they are intended to evaluate. (3) Validation gap: LaaJ has grown rapidly without commensurate human validation (fewer than 8% of papers). Crucially, while LaaJ correlates with aggregate quality, alignment collapses on fine-grained criteria like fluency. To address these gaps, we distill our findings into a minimal Evaluation Checklist to guide metric selection, construct validity, and LaaJ deployment.
Figures & tables
Figure 1: Evaluation method adoption across the 3,334 Top-30 NLG-task papers (2020–2025). Automatic metrics remain ubiquitous ( 95%). LaaJ adoption surged from less than 1% before 2023 to 46% in 2025, surpassing human evaluation, which declined from 51% to 43%. Yet only 16% of papers in 2025 explicitly validate LaaJ against human evaluation, indicating a validation gap.
Figure 2: Our paper annotation pipeline (§ 3 ), including converting PDF to text (§ 3.1 ), extraction of metadata based on four NLG evaluation questions (Table 15 ), harmonization based on majority vote, filtering, and merging (§ 3.2 ), and human validation (§ 4 ), yielding 14k structured papers including a subset of 110 papers with gold annotations.
Figure 3: Per-task evaluation method adoption (%) of top-four NLG tasks per year. The filled area shows that task’s share of all NLG papers each year.
Figure 4: Evaluation trends for the four most frequent NLG tasks (columns) under three evaluation paradigms (rows). Each panel is a separate bump chart: the -axis gives the annual frequency rank (1 = most used) of the eight most common metrics for that task, from 2020 to 2025 on the -axis. Marker size scales with the metric’s likelihood ratio (LR) for the task in that year. Solid lines with filled markers mark a significant overall association with the task; dashed lines with hollow markers mark metrics that are frequent but have a low LR. Endpoint labels give the metric name and (overall LR, ) , where is the number of papers. COMET’s (19.0, 151) in the MT/Automatic Metrics panel thus means it appears in 151 MT papers and is 19 times more likely to appear in MT than elsewhere; it enters at rank 4 in 2021 and holds rank 2 from 2022 onward, behind BLEU. Dotted arcs bridge years in which a metric drops out of the top eight; dashed connectors attach labels to metrics that are outside the top eight in 2025. A missing line in a given year means the metric was not used. Color denotes the QCET L1 dimension: QO (intrinsic to output), QI (relative to input), QT (relative to target reference), QE (relative to external standard), and AUX (QCET extensions).
Figure 5: Within-task co-occurrence LRs for metric–criterion pairs, one panel per task. Rows list metrics and columns list criteria; each axis contains the five most frequent items (black labels) and the five with the highest task-LR (red labels), so the quadrant where frequent metrics meet frequent criteria (yellow background) is distinct from the rest. Each cell is split along the diagonal: the upper-left orange triangle covers the task’s LaaJ papers and the lower-right blue triangle its human-evaluation papers. Colour intensity encodes the co-occurrence LR, and an outline marks a significant pair. In Dialogue Generation, BLEU with Fluency is outlined and shaded on both halves (133 human-evaluation papers, ; 19 LaaJ papers, ), indicating that the pairing is associated with the task under both paradigms, whereas a cell shaded on one side only reflects a paradigm-specific pairing. Across tasks, papers pair metrics and criteria along frequently used terms rather than task-distinctive ones.
Figure 6: Distribution of LaaJ vs. human evaluation scores by metric and criterion. Individual observations are shown as gray dots. denotes the number of studies included. Values are based on human annotated 70 papers.
| Variable (reference) | LR [95% CI] |
| LaaJ use | 1.28 [1.16, 1.41] |
| Year 2021 (vs. 2020) | 1.11 [0.98, 1.27] |
| Year 2022 | 1.08 [0.95, 1.23] |
| Year 2023 | 1.12 [0.99, 1.27] |
| Year 2024 | 0.97 [0.84, 1.11] |
| Year 2025 | 0.86 [0.74, 1.00] |
Table 2: Regression result. Bold: 95% CI excludes 1, significant. Year and venue are exclusive categories, compared to a reference (2020 / ACL); task indicators are not exclusive (a paper can address several tasks), so each task LR compares papers with vs. without that task
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| New leaf id | Name | Origin | Variants | Paper-occ. | Top raw variants (count) |
| QOC-c-2 | Absence of Toxic / Harmful Content | cluster | 95 | 409 | harmfulness (57); safety (53); Safety (48) |
| QEF-w-10 | Engagingness / Interestingness | cluster | 48 | 248 | engagingness (36); Engagement (28); interestingness (27) |
| QEC-c-3 | Absence of Bias / Stereotypes | cluster | 109 | 172 | fairness (9); Bias (7); bias (7) |
| QEF-w-11 | Empathy / Emotional Appropriateness | cluster | 34 | 108 | Empathy (46); empathy (18); Emotional Support (4) |
| QOF-w-6 | Creativity / Non-creativity | frequency | 26 | 157 | creativity (44); Creativity (39); Novelty (18) |
| QEC-w-2 | Refusal Appropriateness | frequency | 45 | 63 | Refusal (8); refusal (6); Abstention (2) |
Table 3: The 6 extension leaves added to QCET (111 117). Origin indicates whether the leaf was discovered by HDBSCAN clustering of stage-1 classification residuals ( cluster ) or by frequency analysis of remaining unclustered residual variants ( frequency ). Variants is the number of distinct raw criterion strings mapped to this leaf; Paper-occ. sums the LaaJ and human paper-occurrences across those variants.
| Term | Cnt | Top Variants (count) |
| Question Answering | 986 | Question Answering (978); question answering (5); Question-Answering (2) |
| Code Generation | 708 | Code Generation (707); code generation (1) |
| Mathematical Reasoning | 219 | Mathematical Reasoning (213); mathematical reasoning (5); Mathematical reasoning (1) |
| Instruction Following | 211 | Instruction Following (196); Instruction-Following (5); instruction following (3) |
| Language Modeling | 163 | Language Modeling (162); language modeling (1) |
| Data To Text Generation | 136 | Data-to-Text Generation (92); Data-to-Text (36); Data to Text (3) |
Table 4: Top-10 normalized tasks and their top variants
| Normalized Term | Cnt | Top Variants (count) |
| BLEU | 2384 | BLEU (2380); BLEU-C (1); BLEU-L (1); BLEU-P (1); BLEU@5 (1) |
| Accuracy | 2248 | Accuracy (1938); accuracy (265); Accuracy@1 (8); Accuracy@k (4); Weighted Accuracy (4) |
| ROUGE | 1838 | ROUGE (1774); ROUGE-L (53); ROUGE-2 (3); ROUGE-S (3); ROUGE-1 (2) |
| F1 | 1548 | F1 (1136); F1-score (116); F1 Score (52); Macro-F1 (45); F1 score (28) |
| BERTScore | 886 | BERTScore (880); BERTSCORE (3); BertScore (3) |
| Recall | 796 | Recall (514); recall (54); Recall@k (44); Recall@10 (32); Recall@K (30) |
Table 5: Top-10 normalized automatic metrics and their top variants
| QCET Target (id) | Cnt | Top Variants (count) |
| Correctness of Outputs (QOC-w-1) | 480 | correctness (173); accuracy (111); Correctness (83) |
| Factual Truth (QEC-c-1) | 465 | Factuality (36); factuality (32); factual consistency (25) |
| Relevance to Input (QIG-c-3) | 438 | relevance (187); Relevance (100); adequacy (7) |
| Coherence (QOG-c-3) | 279 | coherence (93); Coherence (67); logical coherence (8) |
| Consistency with Input (QIC-c-3) | 260 | faithfulness (43); Faithfulness (23); instruction-following (11) |
| Absence of Toxic Content (QOC-c-2) | 241 | harmfulness (42); harmlessness (35); safety (31) |
Table 6: Top-10 LaaJ criteria after QCET-based normalization, with paper count and the most frequent raw variants that mapped to each target.
| QCET Target (id) | Cnt | Top Variants (count) |
| Fluency (QOG-w-3) | 806 | fluency (477); Fluency (269); Language Fluency (6) |
| Relevance to Input (QIG-c-3) | 805 | relevance (280); Relevance (155); adequacy (64) |
| Factual Truth (QEC-c-1) | 778 | factuality (58); Factuality (42); factual consistency (36) |
| Correctness of Outputs (QOC-w-1) | 694 | correctness (211); accuracy (139); Correctness (87) |
| Coherence (QOG-c-3) | 630 | coherence (248); Coherence (163); semantic coherence (8) |
| Overall Quality / Pref. (AUX-OQ) | 499 | overall quality (108); preference (60); quality (46) |
Table 7: Top-10 human evaluation criteria after QCET-based normalization, with paper count and the most frequent raw variants that mapped to each target.
| Term | Cnt | Top Variants (count) |
| Chinese | 1142 | Chinese (1106); Mandarin (18); Simplified Chinese (6) |
| German | 1034 | German (1031); Swiss German (1); Swiss-German (1) |
| Spanish | 537 | Spanish (536); spa (1) |
| Russian | 402 | Russian (401); rus (1) |
| Arabic | 322 | Arabic (288); Egyptian Arabic (13); Modern Standard Arabic (7) |
| Portuguese | 219 | Portuguese (210); Brazilian Portuguese (9) |
Table 8: Top-10 normalized languages and their top variants
| Term | Cnt | Top Variants (count) |
| GSM8K | 454 | GSM8K (428); GSM8k (18); GSM-8K (6) |
| CNN/DailyMail | 303 | CNN/DailyMail (159); CNN/Daily Mail (70); CNN/DM (51) |
| WMT14 | 289 | WMT14 English-German (40); WMT14 (35); WMT14 En-De (26) |
| XSUM | 230 | XSum (182); XSUM (46); Xsum (1) |
| TriviaQA | 223 | TriviaQA (221); TRIVIAQA (1); triviaQA (1) |
| HotpotQA | 214 | HotpotQA (192); HotPotQA (16); HOTPOTQA (6) |
Table 9: Top-10 normalized datasets and their top variants
| Term | Cnt | Top Variants (count) |
| GPT-4 | 918 | GPT-4 (916); ChatGPT (GPT-4) (1); SyncTOD (GPT-4) (1) |
| GPT-4o | 905 | GPT-4o (892); GPT-4O (12); gpt-4o (1) |
| Transformer | 601 | Transformer (594); TRANSFORMER (2); transformer (2) |
| BART | 552 | BART (550); GEE (BART) (1); SOV-MAS (BART) (1) |
| GPT-3.5-turbo | 519 | GPT-3.5-turbo (261); GPT-3.5-Turbo (176); GPT-3.5 Turbo (54) |
| LLaMA-2-7B | 507 | LLaMA-2-7B (126); Llama-2-7B (123); LLaMA2-7B (77) |
Table 10: Top-10 normalized NLG models and their top variants
| Excluded task | Reason |
| Code Generation | Not natural language output; verified by execution, not text quality |
| Mathematical Reasoning | Evaluated by answer correctness against gold answer, not linguistic criteria |
| Math Problem Solving | Same as Mathematical Reasoning |
| Math Reasoning | Same as Mathematical Reasoning |
| Text-To-SQL Generation | Query output; verified by execution accuracy |
| Semantic Parsing | Meaning representation; evaluated by exact/graph match |
Table 11: Excluded Tasks and Reasons
| Top excluded tasks | Top-3 evaluation metrics |
| Code Generation (401) | Accuracy (33%), pass@k (30%), Exact Match (15%) |
| Instruction Following (131) | Accuracy (41%), Win Rate (28%), ROUGE (16%) |
| Math Reasoning, all variants (154) | Accuracy (73%), pass@k (24%), Exact Match (14%) |
| Text-to-SQL Generation (41) | Execution Accuracy (46%), Exact Match Accuracy (32%), Exact Match (24%) |
| Semantic Parsing (55) | Exact Match (36%), Accuracy (29%), Exact Match Accuracy (24%) |
| Language Modeling (102) | Perplexity (78%), Accuracy (36%), BLEU (24%) |
Table 12: Top-3 metrics for newly added tasks
| Field | Mean (1–3) |
| Q1: tasks | 2.62 |
| Q1: datasets | 2.71 |
| Q1: languages | 2.88 |
| Q1: NLG models | 2.52 |
| Q1: output formats | 2.52 |
| Q2: automatic metrics | 2.69 |
Table 13: LLM-extraction field-level Likert scores (60 papers; 1=mismatch, 2=partial, 3=full match). Means above 2.5 across all fields indicate that LLM extraction substantially aligns with human annotators.
| Example A | Example B | |
| (QA exact match) | (coverage rel. factual accuracy) | |
| (papers) | 3,334 | 1,711 |
| ( A and B ) | 136 | 3 |
| ( A , not B ) | 326 | 19 |
| ( B , not A ) | 90 | 25 |
| (neither) | 2,782 | 1,664 |
Table 14: LR and on two real pairs from our data with nearly the same LR but opposite significance outcomes. Example A (task–metric): A = question answering, B = exact match; full set ( ). Example B (metric–criterion): A = coverage, B = relative factual accuracy (QCET QEC-c-2); human-evaluation subset ( ). The large difference in (136 vs. 3) drives the difference in and hence , producing opposite conclusions after BH-FDR correction.
| Key | Description |
| Q1: NLG task presence ( answer_1 ) | |
| answer | “Yes” if the paper addresses any NLG task. |
| quote | Verbatim excerpt supporting the decision. |
| tasks | NLG tasks (e.g., Summarization, MT, Paraphrase Gen, or Other:<task> ). |
| datasets | List of datasets used for generation/evaluation. |
| languages | List of languages (e.g., English, Chinese). |
Table 15: Schema of the JSON object returned by the LLM for each paper.
Figure 7: Heatmap of task-year distribution of the top-30 tasks. Both the total number and percentage per year are shown. The percentage of MT is decreasing while QA-related tasks increase significantly (QA, VQA).
Figure 8: Distribution of metadata by tasks, columns include top-four and all-tasks, and rows are the metadata extracted (counts are after term normalization).
| Task | Criterion | Recommended Metric(s) | Avoid as Primary | LaaJ viable? | Key References |
| Machine Translation | Adequacy / Translation Quality | COMET Rei et al. (2020) , MetricX-25 Juraska et al. (2025) (COMET 39.0% of 2025 MT; BLEU still 61.8%) | BLEU (task-LR , low criterion-LR) | ✓(system-level; segment-level weaker Lavie et al. (2025) ) | Lavie et al. (2025) ; Mathur et al. (2020) |
| Dialogue Generation | Diversity | Distinct- , Self-BLEU Li et al. (2016) | BLEU (task-LR for DG) | ✗(auto metrics sufficient) | Li et al. (2016) |
| Engagement / Appropriateness | LaaJ + human (multi-item Likert) | ROUGE (n-gram overlap) | ✓(rubric) | van der Lee et al. (2019) ; Bavaresco et al. (2025) | |
| Safety / Harmlessness | LaaJ rubric + human red-teaming (47 DG papers use LaaJ with safety criteria) | — | ✓(validate vs. humans; judge: GPT-4/GPT-4o) | Bavaresco et al. (2025) ; Wang et al. (2024) | |
| Text Summarization | Consistency / Faithfulness (vs. input) | SummaC Laban et al. (2022) (28), UniEval Zhong et al. (2022) (23) | ROUGE (low criterion-LR with consistency) | ✓(validation evidence thin, § 5.3 ) | Laban et al. (2022) ; Zhong et al. (2022) |
| Factual Accuracy | QAFactEval Fabbri et al. (2022) (12), MiniCheck Tang et al. (2024) (7, mostly 2024–2025) | ROUGE / BERTScore (similarity factuality) | ✓(validation required, § 5.3 ) | Fabbri et al. (2022) ; Tang et al. (2024) |
Table 19: Task-specific evaluation guidance derived from our empirical analysis (§ 5 ). “Avoid as Primary” lists metrics that co-occur frequently but exhibit low LR with the target criterion, indicating inertial use rather than validated selection. ✓ and ✗ indicate whether an LLM-as-a-judge evaluator is recommended as a viable complement. All recommendations should be validated on the specific task and language before deployment.
More in this topic
Language Model Generation Evaluation
Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.
See all 395 papers