Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study

Thibault Bañeras-Roux; Shashi Kumar; Driss Khalil; Sergio Burdisso; Petr Motlicek; Shiran Liu; Mickael Rouvier; Jane Wottawa

cs.CLAug 26, 2026

Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study

Authors: Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa,

Organizations: Idiap Research Institute, Martigny, Switzerland · LIA, Avignon Universit´e, Avignon, France · LIUM, Le Mans Universit´e, Le Mans, France · LS2N, Nantes Universit´e, Nantes, France

Abstract

Automatic Speech Recognition (ASR) is typically evaluated using Word Error Rate (WER), which poorly reflects semantic similarity. While embedding-based metrics correlate better with human judgments, the respective roles of encoder and decoder-based Large Language Models (LLMs) remain underexplored. This paper presents a comparative study of both families for ASR evaluation. We analyze BERTScore and SemDist across different LLMs, layers, and pooling strategies, showing that both metrics can achieve strong correlation with human judgments when properly configured. For decoder models, we investigate generative LLMs in two settings: pairwise hypothesis selection via prompting and direct qualitative error classification. Our results show that encoder-based metrics remain highly competitive, while generative LLMs perform strongly in hypothesis comparison and improve the interpretability of ASR evaluation.

Figures & tables

Fig. 1 : Agreement with human judgments of BERTScore metric across models and representation layers. Human annotations are obtained from the HATS dataset subset with annotator consensus.

Model	Last	Mean	Mean*	Wgt.	Wgt.*
Gemma-2-2b	73	79	79	79	78
Gemma-2b	76	80	80	79	80
Gemma-3-1b-pt	75	76	74	77	76
Gemma-3-27b-it	77	83	83	81	82
Gemma-4-31B	72	75	73	77	76
Gemma-7b	65	69	71	71	72

TABLE I: Agreement with human judgments of SemDist with different models and pooling strategies, using last-layer representations. Human annotation are obtained from the HATS dataset subset with annotator consensus.

Fig. 2 : Agreement of human judgments of SemDist across encoder and decoder LLMs as a function of layer depth (from the 10th-to-last layer to the last layer). Each cell reports the best pooling strategy. Human annotation are obtained from the HATS dataset subset with annotator consensus.

LLM	$=$ 100 %	$\geq$ 70 %	Full
GPT-4.1	94	85	79
Gemma3-27B	72	63	61
Gemma4-31B	87	78	73
Qwen3.5-35B	92	83	78
Qwen3.5-27B	91	83	77
Qwen3-30B	84	75	71

TABLE II : Agreement of human judgments (HATS) with LLM choices (selecting the best hypothesis), with respect to annotator agreement level.

LLM	Spearman	Pearson
GPT-4.1	-0.66	-0.63
Gemma4-31B	-0.63	-0.63
Qwen3-30B	-0.63	-0.66
Qwen3.5-27B	-0.60	-0.61
Qwen3.5-35B	-0.58	-0.55
Qwen3-1.7B	-0.46	-0.50

TABLE III : Correlation between generated categories and SemDist with embeddings from the last layer of Sentence-CamemBERT-Large using mean pooling.

Fig. 3 : Distribution of SemDist scores across LLM-predicted quality classes.

ASR Evaluation

Automatic speech recognition (ASR) evaluation measures transcript accuracy, robustness, fairness, and usefulness across languages, speakers, and real-world conditions.

See all 145 papers

Evaluation of Automatic Speech Recognition Using Generative Large Language Models
Apr 23, 2026Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil +6ASR Evaluation Automatic Speech Recognition
Idiap Research Institute, Martigny, Switzerland · EPFL, Lausanne, Switzerland · Brno University of Technology, Czech Republic +3
Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER
Sep 18, 2026Hritika Sharma, Thibault Bañeras-Roux, Alessandra Pinto +4ASR Evaluation Semantic Textual Similarity
Faculty of Computer Science and Technology, Algoma University · Idiap Research Institute · School of Psychology, Algoma University +1
Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition
Jul 6, 2026Mohammad Zeineldeen, Albert Zeyer, Haoran Zhang +3ASR Evaluation Automatic Speech Recognition
AppTek.ai GmbH, Aachen, Germany · Machine Learning and Human Language Technology Group, Faculty of Computer Science, RWTH Aachen University Aachen, Germany

Apr 23, 2026J/K move · Enter open · S save

Evaluation of Automatic Speech Recognition Using Generative Large Language Models

Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso+5

Idiap Research Institute, Martigny, Switzerland · EPFL, Lausanne, Switzerland · Brno University of Technology, Czech Republic+3

ASR Evaluation Automatic Speech Recognition

Open paper AskPDF arXiv

Automatic Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning. Embedding-based semantic metrics are better correlated with human perception, but decoder-based Large Language Models (LLMs) remain underexplored for this task. This paper evaluates their relevance through three approaches: (1) selecting the best hypothesis between two candidates, (2) computing semantic distance using generative embeddings, and (3) qualitative classification of errors. On the HATS dataset, the best LLMs achieve 92--94% agreement with human annotators for hypothesis selection, compared to 63% for WER, also outperforming semantic metrics. Embeddings from decoder-based LLMs show performance comparable to encoder models. Finally, LLMs offer a promising direction for interpretable and semantic ASR evaluation.

Abstract

ASR Evaluation

Explore similar work

Evaluation of Automatic Speech Recognition Using Generative Large Language Models

Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER

Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition