Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study
Authors: Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour
Organizations: Idiap Research Institute, Martigny, Switzerland · LIA, Avignon Universit´e, Avignon, France · LIUM, Le Mans Universit´e, Le Mans, France · LS2N, Nantes Universit´e, Nantes, France
Automatic Speech Recognition (ASR) is typically evaluated using Word Error Rate (WER), which poorly reflects semantic similarity. While embedding-based metrics correlate better with human judgments, the respective roles of encoder and decoder-based Large Language Models (LLMs) remain underexplored. This paper presents a comparative study of both families for ASR evaluation. We analyze BERTScore and SemDist across different LLMs, layers, and pooling strategies, showing that both metrics can achieve strong correlation with human judgments when properly configured. For decoder models, we investigate generative LLMs in two settings: pairwise hypothesis selection via prompting and direct qualitative error classification. Our results show that encoder-based metrics remain highly competitive, while generative LLMs perform strongly in hypothesis comparison and improve the interpretability of ASR evaluation.
Figures & tables
Fig. 1 : Agreement with human judgments of BERTScore metric across models and representation layers. Human annotations are obtained from the HATS dataset subset with annotator consensus.
Model
Last
Mean
Mean*
Wgt.
Wgt.*
Gemma-2-2b
73
79
79
79
78
Gemma-2b
76
80
80
79
80
Gemma-3-1b-pt
75
76
74
77
76
Gemma-3-27b-it
77
83
83
81
82
Gemma-4-31B
72
75
73
77
76
Gemma-7b
65
69
71
71
72
TABLE I: Agreement with human judgments of SemDist with different models and pooling strategies, using last-layer representations. Human annotation are obtained from the HATS dataset subset with annotator consensus.
Fig. 2 : Agreement of human judgments of SemDist across encoder and decoder LLMs as a function of layer depth (from the 10th-to-last layer to the last layer). Each cell reports the best pooling strategy. Human annotation are obtained from the HATS dataset subset with annotator consensus.
LLM
= 100 %
≥ 70 %
Full
GPT-4.1
94
85
79
Gemma3-27B
72
63
61
Gemma4-31B
87
78
73
Qwen3.5-35B
92
83
78
Qwen3.5-27B
91
83
77
Qwen3-30B
84
75
71
TABLE II : Agreement of human judgments (HATS) with LLM choices (selecting the best hypothesis), with respect to annotator agreement level.
LLM
Spearman
Pearson
GPT-4.1
-0.66
-0.63
Gemma4-31B
-0.63
-0.63
Qwen3-30B
-0.63
-0.66
Qwen3.5-27B
-0.60
-0.61
Qwen3.5-35B
-0.58
-0.55
Qwen3-1.7B
-0.46
-0.50
TABLE III : Correlation between generated categories and SemDist with embeddings from the last layer of Sentence-CamemBERT-Large using mean pooling.
Fig. 3 : Distribution of SemDist scores across LLM-predicted quality classes.
Automatic Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning. Embedding-based semantic metrics are better correlated with human perception, but decoder-based Large Language Models (LLMs) remain underexplored for this task. This paper evaluates their relevance through three approaches: (1) selecting the best hypothesis between two candidates, (2) computing semantic distance using generative embeddings, and (3) qualitative classification of errors. On the HATS dataset, the best LLMs achieve 92--94% agreement with human annotators for hypothesis selection, compared to 63% for WER, also outperforming semantic metrics. Embeddings from decoder-based LLMs show performance comparable to encoder models. Finally, LLMs offer a promising direction for interpretable and semantic ASR evaluation.
Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metrics tested, that the best-performing SemDist configurations achieve the highest overall agreement, ahead of CER and BERTScore, and that no single model is best across settings. CER, despite its simplicity and low cost, remains remarkably close to these best configurations. In line with prior recommendations, our results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.
Language model (LM) perplexity (PPL) has historically been used as a proxy for automatic speech recognition (ASR) word error rate (WER), with prior work reporting an approximately linear relation in log-log space. Modern end-to-end ASR systems challenge this assumption because they already contain internal language modeling capacity, are often evaluated without external language models, and can now be combined with neural LMs and large language models (LLMs) through different recognition strategies. This paper revisits the relation between PPL and WER for modern ASR systems. We study whether external LMs still improve current end-to-end ASR systems, whether the PPL-WER relation remains linear in log-log space, how encoder context length affects this relation, and how LLM perplexities fit into the trend observed for standard neural LMs. We further investigate internal language modeling (ILM) in attention-based encoder-decoder systems and show that ILM subtraction changes the observed PPL-WER relation, indicating that the decoder's internal LM must be considered when interpreting the effect of external LM quality.
Mohammad Zeineldeen, Albert Zeyer, Haoran Zhang +3
AppTek.ai GmbH, Aachen, Germany · Machine Learning and Human Language Technology Group, Faculty of Computer Science, RWTH Aachen University Aachen, Germany