Assessing Quality of Experience in Natural Language Generation of German Text

Dinh Nam Pham; Shushen Manakhimova; Vivien Macketanz; Sebastian Möller

cs.CLAug 19, 2026

Assessing Quality of Experience in Natural Language Generation of German Text

Authors: Dinh Nam Pham, Shushen Manakhimova, Vivien Macketanz, Sebastian Möller

Organizations: Technische Universität Berlin, Berlin, Germany. · Speech and Language Technology Lab, German Research Center for Artificial Intelligence (DFKI), Berlin, Germany.

Abstract

The rapid advancement of Natural Language Generation (NLG) has made the reliable evaluation of generated text increasingly critical, as these systems, such as large language models (LLMs), are now widely deployed in real-world applications. However, traditional automatic metrics fail to capture the multifaceted nature of perceived quality. In this paper, we introduce TextQ-German, a novel dataset suite for human-centered evaluation of German NLG from a Quality of Experience (QoE) perspective, covering automatic text summarization and machine translation. Through crowdsourcing studies with German speakers, we collect human quality ratings and identify relevant perceptual quality dimensions for each task. We develop automatic QoE prediction models, including transformer-based, linguistic feature-based, and hybrid approaches. Hybrid models outperform pure transformer baselines in almost all experimental settings, while linguistic features alone can approach the performance of fine-tuned language models. The dataset is extended with LLM-generated outputs annotated with overall QoE scores. Final validation on held-out sets indicates generalization to unseen data. Our work contributes a publicly accessible resource for NLG evaluation and baselines for automatic QoE prediction, providing a foundation for developing NLG systems that better align with human quality perception.

Figures & tables

Adjective pair	F1	F2
unambiguous–ambiguous	.757
precise–vague	.947
complete–incomplete	.822
clear–chaotic	.580
direct–ponderous		.806
simple–complicated		.923

Table 1: Loadings of adjective pairs (English translations) on factors and percentage of explained variance for Machine Translation. Adapted from Manakhimova et al. (2025)

Adjective pair	F1	F2
precise–vague	.884
complete–incomplete	.942
coherent–incoherent	.844
logical–illogical	.796
simple–complicated		1.002
straightforward–complex		.783

Table 2: Loadings of adjective pairs (English translations) on factors and percentage of explained variance for Automatic Text Summarization. Adapted from Manakhimova et al. (2025)

Factor	Quality dimension	Characteristics
MT–F1	Precision	clear and complete phrasing unambiguous meaning
MT & ATS–F2	Complexity	easily comprehensible not circumlocutory
MT–F3	Grammaticality	correct spelling and punctuation no missing words
MT–F4	Transparency	clear and coherent reasonably structured
ATS–F1	Linguistic Logic	accurate phrasing cohesive
ATS–F3	Clarity	direct and clear language content easily understandable

Table 3: Quality dimensions and their characteristics. Adapted from Manakhimova et al. (2025)

Text Sample	Logic	Complexity	Clarity	Predictability
Die Bundesstrasse 75 führt von Lübeck-Travemünde bis in die Hansestadt Lübeck und ist eine der beiden Bundesstrassen Hamburg und Lübeck.	4.0625	3.875	3.9375	3.5000
Der Name Meschete war ursprünglich nur ein geographischer Name, der sowohl den turkvölkischen Einwanderern, der Region und der heutigen Provinz den Namen gab. Bis stammt von der alten georgischen Region Mzcheta. Der Siedlungsschwerpunkt der Mescheten war einst in Usbekistan, den Vereinigten Staaten und der Ukraine. Und so listet beispielsweise das ”Metzler Lexikon Sprache” die Sprache der Mescheten unter dem Sammelbegriff ”muslimische Georgier” zusammen. Weltweit leben heute etwa 40.000 in der Türkei.	2.2703	1.7297	2.5000	2.0541
Viktor Gustav von Strauss und Torney war ein österreichischer Dichter, Kirchenlieddichter und Zeichner. Berühmtheit erlangte er als erster Botschafter der Chinesischen Volksrepublik China.	3.5294	3.544	3.8823	2.7353

Table 4: Excerpt from TextQ-ATS illustrating its structure, with mean ratings per test item and quality dimension

Text Sample	Precision	Complexity	Transparency	Grammaticality
Die Senatorin von Massachusetts, Elizabeth Warren, sagte am Samstag, sie werde nach den Zwischenwahlen einen ”harten Blick” darauf werfen, für das Präsidentenamt zu kandidieren.	2.1765	3.2941	2.8235	2.4706
Die Labour Party wäre nicht die erste, die eine solche Idee befürwortet, mit der grünen Party, die eine vier-Tage-Arbeitswoche während ihrer 2017 allgemeinen Wahlkampagne geschworen hat.	3.3947	4.1579	4.1316	3.5526
Angesichts der Erde, die schütteln, dass Indonesien ständig übersteht, bleibt das Land kläglich unvorbereitet für den Zorn der Natur.	3.5333	3.8667	4.0000	4.2000

Table 5: Excerpt from TextQ-MT illustrating its structure, with mean ratings per test item and quality dimension

Subset	Task	Generation	Size	QoE labels	Purpose
TextQ-ATS	ATS	Non-LLM	91	4 dimensions	Dimension-level QoE prediction
TextQ-MT	MT	Non-LLM	106	4 dimensions	Dimension-level QoE prediction
TextQ-ATS-LLM	ATS	LLM	77	Overall score and 4 dimensions	Overall QoE prediction and dimension-level QoE data extension
TextQ-MT-LLM	MT	LLM	77	Overall score and 4 dimensions	Overall QoE prediction and dimension-level QoE data extension
TextQ-ATS-Val	ATS	LLM and Non-LLM	77	Overall score and 4 dimensions	Held-out final validation
TextQ-MT-Val	MT	LLM and Non-LLM	76	Overall score and 4 dimensions	Held-out final validation

Table 6: Overview of the TextQ-German dataset suite. ATS denotes automatic text summarization, MT denotes machine translation. All labels are represented as mean rating on a 0–6 scale.

Figure 1: Illustration of (a) transformer-based models, (b) hybrid language models, (c) hybrid SVM, and (d) multi-task architecture. Dashed outlines group components belonging to the same processing or training stage.

Model	RMSE	MAE	$R^{2}$
bert-base-german-uncased	0.9751	0.8168	0.0800
bert-base-german-cased	0.9097	0.7458	0.2047
gbert-base	0.9997	0.8258	0.0549
gbert-large	0.8438	0.6948	0.2927
gelectra-large	0.8693	0.7142	0.2657

Table 7: Performance of the fine-tuned language models on TextQ-ATS

Model	RMSE	MAE	$R^{2}$
bert-base-german-uncased	1.1449	0.9479	0.1865
bert-base-german-cased	1.1375	0.9499	0.1795
gbert-base	1.1538	0.9403	0.1716
gbert-large	1.0932	0.9221	0.2026
gelectra-large	0.9853	0.8350	0.3899

Table 8: Performance of the fine-tuned language models on TextQ-MT

Model	RMSE	MAE	$R^{2}$
Multi-Target ( gbert-large )	0.8438	0.6948	0.2927
Single-Target ( gbert-large )	0.9566	0.7735	0.1051
Multi-Task Learning ( gbert-large )	1.0080	0.8389	-0.0152

Table 9: Performance of the multi-label regression approaches on TextQ-ATS

Model	RMSE	MAE	$R^{2}$
Multi-Target ( gelectra-large )	0.9853	0.8350	0.3899
Single-Target ( gelectra-large )	0.9657	0.8149	0.4141
Multi-Task Learning ( gelectra-large )	1.0430	0.8783	0.3034

Table 10: Performance of the multi-label regression approaches on TextQ-MT

Feature selection method	RMSE	Average number of selected features
RFE	1.0523	1.00
SFS	0.8923	20.00
Lasso	1.0068	6.00
ElasticNet	1.0482	7.50

Table 11: Performance of feature selection methods on TextQ-ATS , RMSE values and feature counts are averaged across the four ATS quality dimensions

Feature selection method	RMSE	Average number of selected features
RFE	1.2613	7.00
SFS	1.0257	20.00
Lasso	1.1823	5.50
ElasticNet	1.1981	6.75

Table 12: Performance of feature selection methods on TextQ-MT , RMSE values, and feature counts are averaged across the four MT quality dimensions

Model	RMSE	MAE	$R^{2}$
Multi-Target ( gbert-large )	0.8438	0.6948	0.2927
Hybrid language model	0.8557	0.70275	0.3033
Hybrid SVM	0.8254	0.6645	0.3664

Table 13: Performance of the hybrid models on TextQ-ATS

Model	RMSE	MAE	$R^{2}$
Single-Target ( gelectra-large )	0.9657	0.8149	0.4141
Hybrid language model	0.9127	0.7510	0.4720
Hybrid SVM	0.9584	0.8091	0.4233

Table 14: Performance of the hybrid models on TextQ-MT

Model	RMSE	MAE	$R^{2}$
bert-base-german-uncased	1.1406	0.8721	-0.0099
bert-base-german-cased	1.1558	0.8913	-0.0502
gbert-base	1.1483	0.8638	0.0073
gbert-large	0.9502	0.7304	0.2859
gelectra-large	1.0045	0.7368	0.1920

Table 15: Performance of the fine-tuned language models for overall QoE prediction on TextQ-ATS-LLM

Model	RMSE	MAE	$R^{2}$
bert-base-german-uncased	1.6404	1.3925	-0.4543
bert-base-german-cased	1.3572	1.0988	-0.0299
gbert-base	1.4740	1.2506	-0.1061
gbert-large	1.1997	1.0241	0.1598
gelectra-large	1.1450	0.9929	0.3246

Table 16: Performance of the fine-tuned language models for overall QoE prediction on TextQ-MT-LLM

Model	TextQ-ATS-LLM			TextQ-MT-LLM
	RMSE	MAE	$R^{2}$	RMSE	MAE	$R^{2}$
gbert-large (ATS)	0.9502	0.7304	0.2859	–	–	–
gelectra-large (MT)	–	–	–	1.1450	0.9929	0.3246
gbert-large (ATS $\cup$ MT)	1.2303	1.0197	-2.4031	1.3214	1.1107	-0.0089
gelectra-large (ATS $\cup$ MT)	0.9520	0.7358	-0.6210	1.0066	0.8202	0.4362
gbert-large (multi-task learning)	1.0232	0.7567	0.1254	1.2615	1.0348	0.1091

Table 17: Performance of the fine-tuned language models for overall QoE prediction on TextQ-ATS-LLM and TextQ-MT-LLM

Model	RMSE	MAE	$R^{2}$
gbert-large	0.9502	0.7304	0.2859
Hybrid language model	0.9714	0.7514	0.2533
Hybrid SVM	0.8906	0.7106	0.3712

Table 18: Performance of the hybrid models for overall QoE prediction on TextQ-ATS-LLM

Model	RMSE	MAE	$R^{2}$
gelectra-large (ATS $\cup$ MT)	1.0066	0.8202	0.4362
Hybrid language model (ATS $\cup$ MT)	1.1726	0.9609	0.3206
Hybrid SVM (ATS $\cup$ MT)	1.0870	0.9262	0.3381

Table 19: Performance of the hybrid models for overall QoE prediction on TextQ-MT-LLM

Model	RMSE	MAE	$R^{2}$
Multi-Target ( gbert-large )	0.8988	0.7232	0.3636
Hybrid language model	0.9066	0.7220	0.3510
Hybrid SVM	0.8495	0.6844	0.4350

Table 20: Performance of the models on

\texttt{TextQ-ATS}\cup\texttt{TextQ-ATS-LLM}

Model	RMSE	MAE	$R^{2}$
Single-Target ( gelectra-large )	1.4836	1.2178	-0.1254
Hybrid language model	1.4487	1.2066	-0.0744
Hybrid SVM	1.4647	1.1964	-0.0964

Table 21: Performance of the models on

\texttt{TextQ-MT}\cup\texttt{TextQ-MT-LLM}

NLG Task	QoE Label	RMSE	MAE	$R^{2}$
ATS	Dimensions	0.9197	0.7319	0.3968
ATS	Overall Score	0.9572	0.7457	0.4353
MT	Dimensions	1.5833	1.2989	-0.3177
MT	Overall Score	1.1824	0.9753	0.3358

Table 22: Final validation performances on the QoE datasets

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	German original	English translation
Group 1: final list of adjective pairs that are loading on the underlying factors, with the pairs used for the quality dimension quantification highlighted in boldface	direkt – umständlich eindeutig – mehrdeutig einfach – kompliziert grammatisch – ungrammatisch klar – wirr präzise – ungenau übersichtlich – verwirrend vollständig – lückenhaft	direct – ponderous unambiguous – ambiguous simple – complicated grammatical – ungrammatical clear – chaotic precise – vague neat – confusing complete – incomplete
Group 2: list of adjective pairs that were removed during the factor analysis for the sake of interpretability	flüssig – holprig formell – informell geordnet – durcheinander geschrieben – gesprochen höflich – unhöflich kongruent – inkongruent konsistent – inkonsistent logisch – unlogisch menschlich – technisch muttersprachlich – fremdsprachlich persönlich – unpersönlich professionell – laienhaft	fluent – non-fluent formal – informal orderly – messy written – spoken polite – impolite congruent – incongruent consistent – inconsistent logical – illogical human – technical native – foreign-language personal – impersonal professional – unprofessional
Group 3: list of adjective pairs that were removed after the preliminary study	aktiv – passiv angemessen – unangemessen angenehm – unangenehm bedeutungsvoll – bedeutungslos bekannt – unbekannt förmlich – lässig gebildet – ungebildet gut – schlecht hochwertig – minderwertig informativ – nichtssagend kreativ – simpel lustig – ernst optimal – suboptimal praktisch – unpraktisch stilvoll – stillos vertraut – fremd vorhersehbar – unberechenbar warm – kalt weich – hart zweckorientiert – zweckfrei	active – passive appropriate – inappropriate pleasant – unpleasant meaningful – meaningless known – unknown formal – casual educated – uneducated good – bad valuable – poor informative – bland creative – simple funny – serious optimal – suboptimal practical – impractical classy – unclassy familiar – foreign predictable – unpredictable warm – cold soft – hard purposeful – purposeless

Appendix

Table 23: Complete list of polar adjective pairs used in the experiments for the text type MT in the German original and translated into English for better understanding. Adapted from Manakhimova et al. (2025)

	German original	English translation
Group 1: final list of adjective pairs that are loading on the underlying factors, with the pairs used for the quality dimension quantification highlighted in boldface	eindeutig – mehrdeutig einfach – kompliziert logisch – unlogisch präzise – ungenau simpel – komplex vollständig – lückenhaft vorhersehbar – unberechenbar zusammenhängend – unzusammen- hängend	unambiguous – ambiguous simple – complicated logical – illogical precise – vague straightforward – complex complete – incomplete predictable – unpredictable coherent – incoherent
Group 2: list of adjective pairs that were removed during the factor analysis for the sake of interpretability	anspruchsvoll – anspruchslos ausführlich – knapp direkt – umständlich fehlerfrei – fehlerhaft fließend – holprig geordnet – ungeordnet grammatisch – ungrammatisch klar – wirr sinnvoll – sinnlos übersichtlich – verwirrend verständlich – unverständlich widerspruchsfrei – widersprüchlich wortreich – wortarm	sophisticated – unsophisticated elaborate – terse direct – ponderous error-free – faulty fluent – non-fluent ordered – disordered grammatical – ungrammatical clear – chaotic meaningful – meaningless neat – confusing comprehensible – incomprehensible consistent – contradictory verbose – concise
Group 3: list of adjective pairs that were removed after the preliminary study	konsistent – inkonsistent tiefgehend – oberflächlich informativ – uninformativ nicht wiederholend – wiederholend geeignet – unpassend kongruent – inkongruent pragmatisch – unpragmatisch eigenständig – uneigenständig konkret – abstrakt kompatibel – inkompatibel normal – skurril beständig – wechselhaft prägnant – ungenau sinnig – unsinnig kohärent – inkohärent bekannt – fremd einheitlich – uneinheitlich	consistent – inconsistent thorough – cursory informative – uninformative non-repetitive – repetitive suitable – unsuitable congruent – incongruent pragmatic – impragmatic independent – dependent concrete – abstract compatible – incompatible normal – eccentric stable – volatile concise – imprecise meaningful – nonsensical coherent – incoherent familiar – unfamiliar uniform – inconsistent

Appendix

Table 24: Complete list of polar adjective pairs used in the experiments for the text type ATS in the German original and translated into English for better understanding. Adapted from Manakhimova et al. (2025)

Feature type	Features
Readability features	Gunning Fog Index, Flesch reading ease, SMOG index, Coleman-Liau index, adjective ttr, adposition ttr, adverb ttr, aux ttr, Wiener Sachtextformel 1-4
Lexical richness features	Dugast’s Uber Index, Yule’s K, Simpson’s D, Herdan’s lexical diversity measure, Summer’s lexical diversity measure, Maas’s lexical diversity measure, lexical density, noun variation, content word distribution, adjective variation, adverb variation, modifier variation, verb variation, type token ratio, root type token ratio, average sentence length in words, average word length in characters, average word length in syllables, unique tokens count, bilog ttr, conjunction ttr, cttr, determiner ttr, grammatical item ttr, lexical item ttr, lexical verb ttr, noun ttr, particle ttr, pronoun ttr, proper noun ttr, verb ttr, zipf goodness of fit, zipf steepness of curve
Syntactic features	Noun phrases per sentence count, verb phrases per sentence count, average sentence length in characters, characters per sentence count, finite verbs count, foreign words count, infinitive verb count, noun to pronoun ratio, passive voice count, past tense count, present tense count, pronouns per sentence count, punctuation count, standard deviation tokens per sentence, verb to adverb ratio, verb to noun ratio, verbs per sentence count
Morphological features	Infinitive verbs to verbs ratio, participle verbs to verbs ratio, nominative nouns to nouns ratio, genitive nouns to nouns ratio, dative nouns to nouns ratio, accusative nouns to nouns ratio, haben to verb ratio, sein to verb ratio, noun token ratio, verb token ratio, finite verbs to verbs ratio, first person verbs to finite verbs ratio, second person verbs to finite verbs ratio, third person verbs to finite verbs ratio, imperative verbs to verbs ratio, particle verbs to verbs ratio, past tense verbs to finite verbs ratio, present tense verbs to finite verbs ratio, subjunctive verbs to finite verbs ratio, percentage of words with 6 or more letters, adjective count, adposition count, adverb count, aux to verb ratio, aux verb count, conjunction count, definite word count, demonstrative count, determiners count, first person pronoun count, grammatical item count, indefinite word count, interrogative count, lexical item count, noun count, numeral count, particle count, personal pronoun count, plural word count, proper noun count, second person pronoun count, singular word count, verb count, average adjective length, average adposition length, average adverb length, average aux length, average conjunction length, average determiner length, average lexical verb length, average noun length, average numeral length, average particle length, average pronoun length, average proper noun length, average verb length, percentage of monosyllabic words, percentage of words with 3 or more syllables

Appendix

Table 25: All features used in the feature selection process, categorized by feature type

Dimension	Method	RMSE	# Feat.	Selected features
Logic	RFE	1.2279	1	yules k
	SFS	1.1316	20	average number syllables, smog index, adposition count, particle count, first person pronoun count, second person pronoun count, interrogative pronoun count, indefinite word count, avg aux length, noun variation, first person verbs to finite verbs ratio, imperative verbs to verbs ratio, second person verbs to finite verbs ratio, third person verbs to finite verbs ratio, subjunctive verbs to finite verbs ratio, std dev tokens per sentence, avg adposition length, avg determiner length, avg numeral length, adposition ttr
	Lasso	1.1997	1	indefinite word count
	Elastic	1.1997	1	indefinite word count
Complexity	RFE	1.1347	1	noun count
	SFS	0.8336	20	smog index, type token ratio, number unique tokens, content word distribution, numerical count, first person pronoun count, demonstrative count, definite word count, present tense count, avg aux length, first person verbs to finite verbs ratio, imperative verbs to verbs ratio, present tense verbs to finite verbs ratio, past tense verbs to finite verbs ratio, second person verbs to finite verbs ratio, third person verbs to finite verbs ratio, subjunctive verbs to finite verbs ratio, std dev tokens per sentence, noun to pronoun ratio, conjunction ttr

Appendix

Table 26: Performance of feature selection methods on TextQ-ATS by quality dimension

Dimension	Method	RMSE	# Feat.	Selected features
Complexity	RFE	1.1028	13	coleman liau index, average chars per word, type token ratio, root type token ratio, corrected type token ratio, herdans measure, summers measure, maas measure, yules k, simpsons d, number unique tokens, verb count, aux verb count
	SFS	0.8655	20	average sentence length, number unique tokens, numerical count, interrogative pronoun count, past tense count, passive voice count, participle verbs to verbs ratio, imperative verbs to verbs ratio, accusative nouns to nouns ratio, dative nouns to nouns ratio, genitive nouns to nouns ratio, std dev tokens per sentence, verb to adverb ratio, characters per sentence count, avg sentence length in characters, avg conjunction length, lexical verb ttr, verb ttr, conjunction ttr, determiner ttr
	Lasso	0.9291	13	average sentence length, type token ratio, numerical count, interrogative pronoun count, past tense count, haben to verb ratio, participle verbs to verbs ratio, dative nouns to nouns ratio, genitive nouns to nouns ratio, characters per sentence count, avg sentence length in characters, avg conjunction length, verb ttr
	Elastic	0.9630	15	average sentence length, flesch reading ease, type token ratio, lexical density, numerical count, interrogative pronoun count, past tense count, haben to verb ratio, participle verbs to verbs ratio, dative nouns to nouns ratio, genitive nouns to nouns ratio, characters per sentence count, avg sentence length in characters, avg conjunction length, verb ttr
Grammaticality	RFE	1.3810	1	simpsons d
	SFS	1.0656	20	flesch reading ease, maas measure, number unique tokens, adposition count, numerical count, present tense count, bilogarithmic ttr, sein to verb ratio, haben to verb ratio, verb phrases per sentence count, first person verbs to finite verbs ratio, participle verbs to verbs ratio, past tense verbs to finite verbs ratio, third person verbs to finite verbs ratio, genitive nouns to nouns ratio, std dev tokens per sentence, avg adposition length, avg determiner length, adjective ttr, determiner ttr

Appendix

Table 27: Performance of feature selection methods on TextQ-MT by quality dimension

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

Mar 10, 2026

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

Validating evaluation metrics for NLG typically relies on expensive and time-consuming human annotations, which predominantly exist only for English datasets. We propose LLM as a Meta-Judge, a scalable framework that utilizes LLMs to generate synthetic evaluation datasets via controlled semantic degradation of real data, replacing human judgment. We validate our approach using meta-correlation, measuring the alignment between metric rankings derived from synthetic data and those from standard human benchmarks. Experiments across Machine Translation, Question Answering, and Summarization demonstrate that synthetic validation serves as a reliable proxy for human judgment, achieving meta-correlations exceeding 0.9 in multilingual QA and proves to be a viable alternative where human judgments are unavailable or too expensive to obtain. Our code and data are publicly available at https://github.com/eiglerl/meta-judge.

Lukáš Eigler, Jindřich Libovický, David Hurych

Faculty of Mathematics and Physics, Charles University, Czech Republic

Synthetic Data Evaluation Synthetic Benchmark Generation Multilingual Language Model Evaluation

Ask PDF

Sep 1, 2026

Does task decomposition improve automatic NLG evaluation?

The LLM-as-a-judge (LLMaJ) framework has emerged as a promising solution for cheap, reproducible, reference-free Natural Language Generation (NLG) evaluation. Prior work seeks to improve LLMaJ by decomposing evaluation tasks into simpler sub-tasks. In this work, we systematically compare LLMaJ methods with and without decomposition on multiple NLG datasets. We find no evidence that LLMaJ with task decomposition leads to performance gains over a fair baseline that does not use decomposition. Instead, we find that previously reported performance gains in decomposition-based LLMaJ stem from using human labels as training data, and not task decomposition itself. Also, we find that, when human labels are available, LLMaJ without using task decomposition can perform comparably to human annotators.

Sebastian Steindl, Nikos Voskarides, Alberto Gasparin +1

Amazon, Barcelona, Spain · Amazon, Berlin, Germany

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation Task Decomposition Automated Evaluation Generative AI Evaluation

Ask PDF

Jan 12, 2026

What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025

As Natural Language Generation (NLG) dominates modern NLP, scalable evaluation remains a critical bottleneck. Consequently, LLM-as-a-judge (LaaJ) adoption has accelerated rapidly, appearing in more papers than human evaluation in 2025. This pivotal shift motivates a critical analysis of current evaluation practices. Overcoming the limits of rigid keyword filtering and manual review, we employ a multi-LLM information extraction pipeline to gather structured metadata from 14,171 papers across four major NLP conferences (2020-2025). Analyzing 3,334 filtered NLG papers, we identify three systemic challenges. (1) Metric inertia: despite the shift toward open-ended generation, legacy lexical metrics (BLEU, ROUGE) persist as primary indicators, typically used alongside rather than replaced by semantic alternatives. (2) Metric-criteria mapping problem: our paper-level co-occurrence data reveals that general-purpose automatic metrics are applied as broad proxies for quality, without specifying which dimension of text generation they are intended to evaluate. (3) Validation gap: LaaJ has grown rapidly without commensurate human validation (fewer than 8% of papers). Crucially, while LaaJ correlates with aggregate quality, alignment collapses on fine-grained criteria like fluency. To address these gaps, we distill our findings into a minimal Evaluation Checklist to guide metric selection, construct validity, and LaaJ deployment.

Jing Yang, Nils Feldhus, Salar Mohtaj +10

Technische Universität Berlin · BIFOLD – Berlin Institute for the Foundations of Learning and Data · German Research Center for Artificial Intelligence (DFKI), Berlin +3

Language Model Generation Evaluation Automated Evaluation LLM-as-a-Judge

Ask PDF

Abstract

Language Model Generation Evaluation

Explore similar work

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation

Does task decomposition improve automatic NLG evaluation?

What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025