Organizations: Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands · Penobscot Community Health Care, United States · Department of Computational Linguistics, University of Zurich, Switzerland
Personalizing automatic speech recognition (ASR) systems for non-normative speech, such as dysarthric and aphasic speech, is challenging. While speaker-specific fine-tuning (SS-FT) is widely used, it is typically initialized directly from a generic pre-trained model. Whether speaker-independent adaptation provides a stronger initialization prior under such mismatch remains unclear. In this work, we propose a two-stage adaptation framework consisting of speaker-independent fine-tuning (SI-FT) on multi-speaker non-normative data followed by SS-FT, and evaluate it through a controlled comparison with direct SS-FT under identical per-speaker conditions. Experiments on AphasiaBank and UA-Speech with Whisper-Large-v3 and Qwen3-ASR, alongside evaluation on typical-speech datasets TED-LIUM v3 and FLEURS, show that two-stage adaptation consistently improves personalization while maintaining manageable out-of-domain (OOD) trade-offs.
Figures & tables
Figure 1: Overview of our methodology. Left: transcript normalization and filtering for AphasiaBank. Middle: speaker-level partition into speaker-independent (SI) and speaker-specific (SS) groups (top 10% speakers selected for personalization), with per-speaker train/validation/test splits. Right: comparison of four experimental conditions (B1–B4), isolating the effect of speaker-independent initialization for speaker-specific adaptation.
\multirow 2*Setup
B1
B2
B3
B4
WER
CER
WER
CER
WER
CER
WER
CER
\cellcolor AphBG AphasiaBank
SI-FT
39.97
31.18
27.70
21.97
–
–
–
–
SS-FT
29.39
23.47
23.90
19.59
18.05
14.58
20.11
16.34
FLEURS
4.17
1.85
4.74
2.24
5.20
2.56
4.70
2.25
TED-LIUM v3
4.00
2.60
2.87
1.71
3.14
1.91
3.08
1.80
Table 1: WER and CER for SI-FT and SS-FT on non-normative AphasiaBank and UA-Speech datasets, and OOD evaluation on typical-speech benchmarks (FLEURS and TED-LIUM v3) under four adaptation settings (B1, B2, B3, B4).
Personalizing Automatic Speech Recognition (ASR) for non-normative speech remains challenging because data collection is labor-intensive and model training is technically complex. To address these limitations, we propose Adapt4Me, a web-based decentralized environment that operationalizes Bayesian active learning to enable end-to-end personalization without expert supervision. The app exposes data selection, adaptation, and validation to lay users through a three-stage human-in-the-loop workflow: (1) rapid profiling via greedy phoneme sampling to capture speaker-specific acoustics; (2) backend personalization using Variational Inference Low-Rank Adaptation (VI-LoRA) to enable fast, incremental updates; and (3) continuous improvement, where users guide model refinement by resolving visualized model uncertainty via low-friction top-k corrections. By making epistemic uncertainty explicit, Adapt4Me reframes data efficiency as an interactive design feature rather than a purely algorithmic concern. We show how this enables users to personalize robust ASR models, transforming them from passive data sources into active authors of their own assistive technology.
Niclas Pokel, Yiming Zhao, Pehuén Moure +2
Institute of Neuroinformatics, University of Zurich and ETH Zurich · Department of Computer Science, ETH Zurich · Department of Computational Linguistics, University of Zurich
Automatic speech recognition (ASR) systems often perform poorly in dysarthric speech, limiting their usefulness to affected speakers in everyday communication. This paper presents a personalized ASR system for a dysarthric speaker, built by adapting a foundation ASR model to speaker-specific data. Using the TEQST tool, we collected 92 hours of read speech and later added 8.8 hours of user corrections gathered through a deployed mobile application. Starting from Whisper, fine-tuning reduced word error rate to 15.8% with only 1.4 hours of adaptation data, reached 10.7% with 22.5 hours, and achieved the best result of 9.7% when using all available data including the corrections. Using LoRA adaptation and/or Qwen3-ASR as foundation model performed worse in this setting. The results show that personalized fine-tuning can make foundation ASR models substantially more effective for dysarthric speech and suitable for practical deployment.
Christian Huber, Laura Kernahan, Alexander Waibel
Interactive Systems Lab Karlsruhe Institute of Technology Karlsruhe, Germany · Interactive Systems Lab Carnegie Mellon University Pittsburgh PA, USA
Automatic Speech recognition (ASR) struggles with Pathological speech due to persistent speaker-specific mismatch. In clinical and accessibility scenarios, only limited target-speaker data is typically available, making full model retraining impractical. We present a component-level analysis of LoRA adaptation in Whisper, investigating how adapter placement interacts with the model size, training efficiency and utterance-duration in single-speaker settings. Our experimental comparative study shows that while scaling reduces errors, it does not eliminate mismatch. Furthermore, the full attention LoRa provides the most reliable accuracy, and the best time--accuracy trade-off. Improvements vary across utterance length, with medium-length recordings benefiting the most, while short and long utterances remain challenging. These findings highlight the importance of adaptation placement and duration-dependent evaluation, under challenging, data-scarce personalization scenarios.
Giuseppe Giaimis, Sergei Katkov, Muhammad Azfar Yaqub +1