Organizations: Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands · Penobscot Community Health Care, United States · Department of Computational Linguistics, University of Zurich, Switzerland
Personalizing automatic speech recognition (ASR) systems for non-normative speech, such as dysarthric and aphasic speech, is challenging. While speaker-specific fine-tuning (SS-FT) is widely used, it is typically initialized directly from a generic pre-trained model. Whether speaker-independent adaptation provides a stronger initialization prior under such mismatch remains unclear. In this work, we propose a two-stage adaptation framework consisting of speaker-independent fine-tuning (SI-FT) on multi-speaker non-normative data followed by SS-FT, and evaluate it through a controlled comparison with direct SS-FT under identical per-speaker conditions. Experiments on AphasiaBank and UA-Speech with Whisper-Large-v3 and Qwen3-ASR, alongside evaluation on typical-speech datasets TED-LIUM v3 and FLEURS, show that two-stage adaptation consistently improves personalization while maintaining manageable out-of-domain (OOD) trade-offs.
Figures & tables
Figure 1: Overview of our methodology. Left: transcript normalization and filtering for AphasiaBank. Middle: speaker-level partition into speaker-independent (SI) and speaker-specific (SS) groups (top 10% speakers selected for personalization), with per-speaker train/validation/test splits. Right: comparison of four experimental conditions (B1–B4), isolating the effect of speaker-independent initialization for speaker-specific adaptation.
\multirow 2*Setup
B1
B2
B3
B4
WER
CER
WER
CER
WER
CER
WER
CER
\cellcolor AphBG AphasiaBank
SI-FT
39.97
31.18
27.70
21.97
–
–
–
–
SS-FT
29.39
23.47
23.90
19.59
18.05
14.58
20.11
16.34
FLEURS
4.17
1.85
4.74
2.24
5.20
2.56
4.70
2.25
TED-LIUM v3
4.00
2.60
2.87
1.71
3.14
1.91
3.08
1.80
Table 1: WER and CER for SI-FT and SS-FT on non-normative AphasiaBank and UA-Speech datasets, and OOD evaluation on typical-speech benchmarks (FLEURS and TED-LIUM v3) under four adaptation settings (B1, B2, B3, B4).
Institute of Neuroinformatics, University of Zurich and ETH Zurich · Department of Computer Science, ETH Zurich · Department of Computational Linguistics, University of Zurich