eess.ASFeb 25, 2026

Quality of Automatic Speech Recognition -- Polish Language case study -- from Wav2Vec to Scribe ElevenLabs

Authors: Marcin Pietroń, Szymon Piórkowski, Kamil Faber, Dominik Żurek, Michał Karwatowski, Jerzy Duda, Hubert Zieliński, Piotr Lipnicki, 

Organizations: Institute of Electronics AGH, Krakow · Coraz Zdrowiej, Krakow · Department of Computer Science AGH, Krakow · Department of Management AGH, Krakow · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This article concerns comparative studies on the Automatic Speech Recognition (ASR) model incorporated with the Large Language Model (LLM) used for medical interviews. The proposed solution is tested on polish language benchmarks and dataset with medical interviews. The latest ASR technologies are based on convolutional neural networks (CNNs), recurrent neural networks (RNNs) and Transformers. Most of them work as end-to-end solutions. The presented approach in the case of the Whisper model shows a two-stage solution with End-To-End ASR and LLM working together in a pipeline. The ASR output is an input for LLM. The LLM is a component by which the output from ASR is corrected and improved. Comparative studies for automatic recognition of the Polish language between modern End-To-End deep learning architectures and the ASR hybrid model were performed. The medical interview tests were performed with two state-of-the-art ASR models: OpenAI Whisper incorporated with LLM and Scribe ElevenLabs. Additionally, the results were compared with five more end-to-end models (QuartzNet, FastConformer, Wav2Vec 2.0 XLSR and ESPnet Model Zoo) on Mozilla Common Voice and VoxPopuli databases. Tests were conducted for clean audio signal, signal with bandwidth limitation, and degraded. The tested models were evaluated on the basis of Word Error Rate (WER) and Character Error Rate (CER). The results show that the Whisper model performs by far the best among the open-source models. ElevenLabs Scribe model, on the other hand, performs best for Polish on both general benchmark and medical data.

Figures & tables

Figure 1: ASR architecture
Figure 2: CORA AI architecture
ModelWER
ElevenLabs6.77%
Whisper large5.05%
Wav2Vec17.2%
Table 1: Comparison between different ASR models - LibriSpeech
 all wordswrong wordscorrect sentenceswrong sentencesall sentencesWERF1 wordsF1 sentences
ElevenLabs16 3243442632575206.7795.850.57
Whisper large17 14075717217 0345205.0595.8733.07
Whisper medium17 0841 0611164045206.8493.9222.3
Whisper small17 0711 8104347752011.9289.498.26
Table 2: Comparison between different Whisper models and ElevenLabs for LibriSpeech
 all wordswrong wordscorrect sentenceswrong sentencesall sentencesWERF1 wordsF1 sentences
ElevenLabs975 14533 166110 82621 570132 3963.2997.3183.7
Whisper large975 14559 48794 86737 529132 3966.394.5471.65
Whisper medium975 145104 30375 75656 640132 39611.6589.7657.21
Whisper small975 145167 82754 10878 288132 39619.2883.0440.86
Table 3: Comparison between different Whisper models and ElevenLabs for CommonVoice
ModelWER
ElevenLabs3.29%
Whisper large6.3%
QuartzNet14%
FastConf5.99%
ESPnet215.1%
Wav2Vec9.8%
Table 4: Comparison between different ASR models - Common Voice
qualityElevenLabsWhisper
WER raw12.0653.83
WER norm10.5842.26
CER raw8.3432.14
CER norm7.9828.58
Table 5: ElevenLabs and Whisper comparison on medical interviews
More in this topic

Medical Speech Recognition

See all 10 papers

Explore similar work

CardsList
Jul 6, 2026

Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition

Language model (LM) perplexity (PPL) has historically been used as a proxy for automatic speech recognition (ASR) word error rate (WER), with prior work reporting an approximately linear relation in log-log space. Modern end-to-end ASR systems challenge this assumption because they already contain internal language modeling capacity, are often evaluated without external language models, and can now be combined with neural LMs and large language models (LLMs) through different recognition strategies. This paper revisits the relation between PPL and WER for modern ASR systems. We study whether external LMs still improve current end-to-end ASR systems, whether the PPL-WER relation remains linear in log-log space, how encoder context length affects this relation, and how LLM perplexities fit into the trend observed for standard neural LMs. We further investigate internal language modeling (ILM) in attention-based encoder-decoder systems and show that ILM subtraction changes the observed PPL-WER relation, indicating that the decoder's internal LM must be considered when interpreting the effect of external LM quality.

Mohammad Zeineldeen, Albert Zeyer, Haoran Zhang +3

AppTek.ai GmbH, Aachen, Germany · Machine Learning and Human Language Technology Group, Faculty of Computer Science, RWTH Aachen University Aachen, Germany

ASR EvaluationAutomatic Speech Recognition
AskPDF
Aug 19, 2026

Understanding Multilingual Medical ASR Adaptation Through Layer-Wise Analysis

Medical automatic speech recognition (MedASR) requires adaptation to specialised terminology, limited annotated clinical data, and multilingual use cases. Although large-scale pretrained ASR models such as Whisper achieve strong generalisation, their behaviour after medical and multilingual adaptation remains insufficiently understood beyond word error rate (WER). This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis. We compare zero-shot decoding, English-only fine-tuning, German-only diagnostic fine-tuning, two-stage EN->EN+DE continuation, and direct EN+DE fine-tuning across Whisper model sizes. Fine-tuning substantially improves MedASR performance, but the best model depends on the adaptation setting: Whisper-Medium gives the lowest English WER (7.72%) and the lowest combined EN+DE WER under direct EN+DE training (26.30%); German-only Whisper-Large-v3 gives the lowest German WER (44.96%), but as a within-corpus diagnostic on 86 single-speaker training utterances rather than robust generalisation. Layer-wise analysis of the two-stage Whisper-Small trajectory shows that English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space. Domain and language information remain highly recoverable across layers, while linearly recoverable error-predictive cues weaken as WER improves.

Souranil Kahali, Rituparna Bose, Abner Hernandez +4

Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen, Germany · School of Computer Science, University of Sheffield, Sheffield, UK · Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM), Munich, Germany

Medical Speech RecognitionAutomatic Speech RecognitionDomain Adaptation for ASRMultilingual ASRRepresentation Learning
AskPDF
May 9, 2026

WASIL: In-the-Wild Arabic Spoken Interactions with LLMs

Large Language Models (LLMs) voice assistants are commonly built as cascaded Automatic Speech recognition (ASR) to LLM systems, where recognition errors can distort user intent. Dislikes may also arise from ambiguous, out-of-domain, or non-request turns, making it hard to isolate ASR effects. We release WASIL (it denotes connection or linking in Arabic): in-the-wild Arabic spoken interaction prompts with audio, ASR hypotheses, assistant responses, and explicit like/dislike feedback (8,529 turns; 14.2% dislikes), plus a 2,000-turn test set covering Modern Standard Arabic (MSA) and four major dialects with their labels. We provide low-cost gold transcripts via multi-ASR agreement-guided post-editing and annotate answerability (answerable, ambiguous/needs-clarification, unsupported, not-a-request/noise) to separate intrinsic unanswerability from ASR-induced degradation. Finally, we describe scalable reference-free evaluation of responses from ASR vs. gold transcripts using multi-judge LLM scoring.

Zien Sheikh Ali, Hamdy Mubarak, Soon-Gyo Jung +3

Qatar Computing Research Institute, Qatar

Voice Agent EvaluationArabic NLPAutomatic Speech RecognitionSpeech Language ModelsLanguage Model Robustness
AskPDF