This work presents an automatic speech recognition (ASR) system personalized for a Czech speaker with a permanent tracheal stoma and severe dysarthria rendering their speech unintelligible to untrained listeners. We release a public dataset containing 33 annotated hours of the speaker's speech, collected using a novel "artificial conversation" protocol designed for high engagement and dialogue realism. We propose a multi-stage training pipeline based on Whisper Base: fine-tuning on standard Czech speech, acoustically simulated tracheostomic speech, and the speaker's data. We evaluate the system across three near real-time scenarios: scripted conversations, question answering, and spontaneous dialogue, achieving a 50% relative reduction in Character Error Rate compared to Whisper Base baseline and surpassing the average recognition accuracy of their assistants in acoustic recognition of isolated utterances. We demonstrate that even for severely impeded speech, a helpful ASR is achievable, as evidenced by the quantitative results and the feedback from the speaker.
Figures & tables
Fig. 1: Speech disorders of the target speaker - Mel spectrograms (128 bins) comparing their speech to that of a standard speaker.
Fig. 2: Artificial conversations - the upper track depicts the synthesized audio guide provided to the speaker. The lower track illustrates the silence windows where the speaker was expected to repeat their line of the scripted dialogue.
Session Type
Sessions
Hours
Utts
Artif. Convs. (Offline)
69
20:21
8 428
Artif. Convs. (Streaming)
30
10:39
2 733
Reading
21
3:06
2 719
Question Answering
3
0:48
353
Spontaneous Convs.
4
1:35
737
Other (Singing, etc.)
10
1:16
454
TABLE I: Overall Dataset Statistics
Session Type
Sessions
Hours
Utts
Artif. Convs
3
1:08
329
Question Answering
3
0:48
353
Spontaneous Convs.
1
0:23
188
TABLE II: Test Set Statistics (Subset of the Overall Dataset)
Fig. 3: KD mechanism.
Evaluation Setup
Scripted
QA
Spont.
Non-impaired Control Data
Baseline
0.289
0.286
0.710
Non-impaired Topline (Czech FT)
0.197
0.172
0.256
Target Speaker Data (Impeded)
Baseline (Zero-shot)
0.944
0.882
1.529
Speaker-only
0.431
0.510
0.949
TABLE III: Comparison of CER across test scenarios. ”Baseline” is off-the-shelf Whisper Base. ”Non-impaired Topline” represents the theoretical upper bound performance: the Whisper Base model fine-tuned by us on standard Czech speech (Common Voice and Ortofon). ”Speaker-only” stands for a model fine-tuned only on the speaker’s speech without the pre-training stages.
Configuration
Scripted
QA
Spont.
Mean
Avg. Rel. Δ
Full System
0.433
0.449
0.637
0.506
Reference
Training Components
w/o Device Adaptation
0.637
0.637
0.696
0.657
+32.7%
w/ Stretch-only Ac. Simulation
0.411
0.437
0.869
0.572
+9.6%
w/o Acoustic Simulation
0.453
0.500
0.647
0.533
+5.8%
w/o Common Voice Pre-train
0.471
0.475
0.645
0.530
+5.2%
TABLE IV: Ablation study. Results reported in CER. The Avg. Rel. Δ column represents the average percentage increase in error relative to the Full System (higher implies the component is more critical). The best result in each column is bolded , and the second best is underlined .
Training Stage (Cumulative)
Scripted
QA
Spont.
Mean
Whisper Base (Zero-shot)
0.944
0.882
1.529
1.118
+ Czech FT (Standard Speech)
0.904
1.796 *
2.633 *
1.778 *
+ Simulated Tracheostomic Speech
0.772
1.082 *
5.358 *
2.404 *
+ Offline Speaker Data
0.637
0.637
0.696
0.657
+ Streaming Data (Full System)
0.433
0.449
0.637
0.506
* Values inflated heavily by unconstrained insertion errors (hallucinations) prior to speaker-specific grounding.
TABLE V: Incremental evaluation across the multi-stage training pipeline. Results are reported in Character Error Rate (CER).
Reference (Ground Truth)
Hypothesis (System Output)
(Translation provided below)
Ahoj, Šalino
Ahoj, Šalino
Hi, Šalina
Hi, Šalina
Lidé zapomínají na oblohu
Lidé zapomínají na oblohu
People forget about the sky
People forget about the sky
Noční oblohu
Naští oblohu
TABLE VI: Qualitative analysis of a sample transcript. Errors are marked in red .