ASR Evaluation

ASR: Automatic Speech Recognition

Overview

ASR evaluation studies how well speech systems turn audio into text and whether their errors matter for people and downstream tasks. Word error rate and character error rate remain common, but researchers also use language-aware normalization, semantic and human-preference measures, and task-based metrics for details such as speaker attribution, timestamps, or critical terms. Benchmarks increasingly test diverse languages and dialects, conversational overlap, clinical or other specialized speech, and acoustic shifts such as noise and reverberation. A central tension is that standardized scores enable comparisons, yet a single reference transcript or aggregate error rate can misrepresent valid linguistic variation, fairness, and practical impact.

Latest papers

151 papers13 papers in the last four weeks, down 55% on the four weeks before. 0.1% of all new papers.

Feb 25, 2026

Quality of Automatic Speech Recognition -- Polish Language case study -- from Wav2Vec to Scribe ElevenLabs

This article concerns comparative studies on the Automatic Speech Recognition (ASR) model incorporated with the Large Language Model (LLM) used for medical interviews. The proposed solution is tested on polish language benchmarks and dataset with medical interviews. The latest ASR technologies are based on convolutional neural networks (CNNs), recurrent neural networks (RNNs) and Transformers. Most of them work as end-to-end solutions. The presented approach in the case of the Whisper model shows a two-stage solution with End-To-End ASR and LLM working together in a pipeline. The ASR output is an input for LLM. The LLM is a component by which the output from ASR is corrected and improved. Comparative studies for automatic recognition of the Polish language between modern End-To-End deep learning architectures and the ASR hybrid model were performed. The medical interview tests were performed with two state-of-the-art ASR models: OpenAI Whisper incorporated with LLM and Scribe ElevenLabs. Additionally, the results were compared with five more end-to-end models (QuartzNet, FastConformer, Wav2Vec 2.0 XLSR and ESPnet Model Zoo) on Mozilla Common Voice and VoxPopuli databases. Tests were conducted for clean audio signal, signal with bandwidth limitation, and degraded. The tested models were evaluated on the basis of Word Error Rate (WER) and Character Error Rate (CER). The results show that the Whisper model performs by far the best among the open-source models. ElevenLabs Scribe model, on the other hand, performs best for Polish on both general benchmark and medical data.