This work presents an automatic speech recognition (ASR) system personalized for a Czech speaker with a permanent tracheal stoma and severe dysarthria rendering their speech unintelligible to untrained listeners. We release a public dataset containing 33 annotated hours of the speaker's speech, collected using a novel "artificial conversation" protocol designed for high engagement and dialogue realism. We propose a multi-stage training pipeline based on Whisper Base: fine-tuning on standard Czech speech, acoustically simulated tracheostomic speech, and the speaker's data. We evaluate the system across three near real-time scenarios: scripted conversations, question answering, and spontaneous dialogue, achieving a 50% relative reduction in Character Error Rate compared to Whisper Base baseline and surpassing the average recognition accuracy of their assistants in acoustic recognition of isolated utterances. We demonstrate that even for severely impeded speech, a helpful ASR is achievable, as evidenced by the quantitative results and the feedback from the speaker.
Figures & tables
Fig. 1: Speech disorders of the target speaker - Mel spectrograms (128 bins) comparing their speech to that of a standard speaker.
Fig. 2: Artificial conversations - the upper track depicts the synthesized audio guide provided to the speaker. The lower track illustrates the silence windows where the speaker was expected to repeat their line of the scripted dialogue.
Session Type
Sessions
Hours
Utts
Artif. Convs. (Offline)
69
20:21
8 428
Artif. Convs. (Streaming)
30
10:39
2 733
Reading
21
3:06
2 719
Question Answering
3
0:48
353
Spontaneous Convs.
4
1:35
737
Other (Singing, etc.)
10
1:16
454
TABLE I: Overall Dataset Statistics
Session Type
Sessions
Hours
Utts
Artif. Convs
3
1:08
329
Question Answering
3
0:48
353
Spontaneous Convs.
1
0:23
188
TABLE II: Test Set Statistics (Subset of the Overall Dataset)
Fig. 3: KD mechanism.
Evaluation Setup
Scripted
QA
Spont.
Non-impaired Control Data
Baseline
0.289
0.286
0.710
Non-impaired Topline (Czech FT)
0.197
0.172
0.256
Target Speaker Data (Impeded)
Baseline (Zero-shot)
0.944
0.882
1.529
Speaker-only
0.431
0.510
0.949
TABLE III: Comparison of CER across test scenarios. ”Baseline” is off-the-shelf Whisper Base. ”Non-impaired Topline” represents the theoretical upper bound performance: the Whisper Base model fine-tuned by us on standard Czech speech (Common Voice and Ortofon). ”Speaker-only” stands for a model fine-tuned only on the speaker’s speech without the pre-training stages.
Configuration
Scripted
QA
Spont.
Mean
Avg. Rel. Δ
Full System
0.433
0.449
0.637
0.506
Reference
Training Components
w/o Device Adaptation
0.637
0.637
0.696
0.657
+32.7%
w/ Stretch-only Ac. Simulation
0.411
0.437
0.869
0.572
+9.6%
w/o Acoustic Simulation
0.453
0.500
0.647
0.533
+5.8%
w/o Common Voice Pre-train
0.471
0.475
0.645
0.530
+5.2%
TABLE IV: Ablation study. Results reported in CER. The Avg. Rel. Δ column represents the average percentage increase in error relative to the Full System (higher implies the component is more critical). The best result in each column is bolded , and the second best is underlined .
Training Stage (Cumulative)
Scripted
QA
Spont.
Mean
Whisper Base (Zero-shot)
0.944
0.882
1.529
1.118
+ Czech FT (Standard Speech)
0.904
1.796 *
2.633 *
1.778 *
+ Simulated Tracheostomic Speech
0.772
1.082 *
5.358 *
2.404 *
+ Offline Speaker Data
0.637
0.637
0.696
0.657
+ Streaming Data (Full System)
0.433
0.449
0.637
0.506
* Values inflated heavily by unconstrained insertion errors (hallucinations) prior to speaker-specific grounding.
TABLE V: Incremental evaluation across the multi-stage training pipeline. Results are reported in Character Error Rate (CER).
Reference (Ground Truth)
Hypothesis (System Output)
(Translation provided below)
Ahoj, Šalino
Ahoj, Šalino
Hi, Šalina
Hi, Šalina
Lidé zapomínají na oblohu
Lidé zapomínají na oblohu
People forget about the sky
People forget about the sky
Noční oblohu
Naští oblohu
TABLE VI: Qualitative analysis of a sample transcript. Errors are marked in red .
Automatic speech recognition (ASR) systems often perform poorly in dysarthric speech, limiting their usefulness to affected speakers in everyday communication. This paper presents a personalized ASR system for a dysarthric speaker, built by adapting a foundation ASR model to speaker-specific data. Using the TEQST tool, we collected 92 hours of read speech and later added 8.8 hours of user corrections gathered through a deployed mobile application. Starting from Whisper, fine-tuning reduced word error rate to 15.8% with only 1.4 hours of adaptation data, reached 10.7% with 22.5 hours, and achieved the best result of 9.7% when using all available data including the corrections. Using LoRA adaptation and/or Qwen3-ASR as foundation model performed worse in this setting. The results show that personalized fine-tuning can make foundation ASR models substantially more effective for dysarthric speech and suitable for practical deployment.
Christian Huber, Laura Kernahan, Alexander Waibel
Interactive Systems Lab Karlsruhe Institute of Technology Karlsruhe, Germany · Interactive Systems Lab Carnegie Mellon University Pittsburgh PA, USA
In our goal to develop personalised dysarthric speech recognition (DSR) models, this study compared the recognition performances of human listeners and those of three state-of-the-art, off-the-shelf ASR systems (Whisper-large-V3, Google Chirp 3, and Omnilingual) on the recognition of Dutch continuous read and spontaneous speech from a single speaker with severe dysarthria. Results showed that both humans listeners and the three off-the-shelf ASR systems exhibit word error rates (WER) exceeding 70% on average, indicating that DSR is highly challenging for both humans and ASR systems. Fine-tuning on the dysarthric speech significantly reduced WER. Although overall WERs are still quite high (>23%), the personalised DSR models outperformed the human listeners, and performance is getting closer to being useful for supporting day-to-day communication of dysarthric speakers. Future research should focus on improving personalized DSR on spontaneous speech and longer utterances in the case of read speech, with a specific focus on particular phonemes.
Yuanyuan Zhang, Dimme de Groot, Jorge Martinez +1
Multimedia Computing Group Delft University of Technology Delft, the Netherlands
Individuals with dysarthria face significant challenges in professional speaking scenarios such as conferences, presentations, and meetings, where real-time communication is crucial. While existing Augmentative and Alternative Communication (AAC) systems provide basic support, they often fail to meet the demands of professional speaking environments due to high latency and unnatural speech patterns. This paper presents Re-Sonance, a novel LLM-enhanced speech-driven AAC system designed for real-time professional speaking scenarios. By integrating Whisper ASR, Qwen LLM, and CosyVoice TTS, Re-Sonance achieves improved speech intelligibility and naturalness while maintaining real-time performance. Both subjective and objective evaluations using a Mandarin dysarthric speech dataset demonstrate that our speech reconstruction approach significantly improved intelligibility while preserving semantic coherence for speakers with mild to moderate dysarthria. Although performance remains limited for severe dysarthria cases, our findings validate the potential of LLM-based methods for enhancing speech-driven AAC systems, paving the way for more effective and accessible communication technologies.
Yuxuan Wu, Yifan Xu, Junkun Wang +3
State Key Laboratory of Digital Medical Engineering, Southeast University, Nanjing, China