Organizations: Institute of Neuroinformatics, University of Zurich and ETH Zurich · Department of Computer Science, ETH Zurich · Department of Computational Linguistics, University of Zurich
Personalizing Automatic Speech Recognition (ASR) for non-normative speech remains challenging because data collection is labor-intensive and model training is technically complex. To address these limitations, we propose Adapt4Me, a web-based decentralized environment that operationalizes Bayesian active learning to enable end-to-end personalization without expert supervision. The app exposes data selection, adaptation, and validation to lay users through a three-stage human-in-the-loop workflow: (1) rapid profiling via greedy phoneme sampling to capture speaker-specific acoustics; (2) backend personalization using Variational Inference Low-Rank Adaptation (VI-LoRA) to enable fast, incremental updates; and (3) continuous improvement, where users guide model refinement by resolving visualized model uncertainty via low-friction top-k corrections. By making epistemic uncertainty explicit, Adapt4Me reframes data efficiency as an interactive design feature rather than a purely algorithmic concern. We show how this enables users to personalize robust ASR models, transforming them from passive data sources into active authors of their own assistive technology.
Figures & tables
Figure 1. Adapt4Me : an end-to-end, human-in-the-loop ASR personalization system for non-normative speech . ① The system prompts the user with high-information sentences for initial recordings; ② Word-level model uncertainty is visualized through color-coding, guiding users directly to likely identify and correct transcribing errors; ③ Users resolve errors via low-effort top- k best corrections powered by a Bayesian backend, with validated edits continuously fed back into active learning and model fine-tuning. A three-panel screenshot of the web application. The left panel shows a microphone icon and a prompt. The center panel shows transcribed text with the word ``Pellikan'' highlighted in red. The right panel shows a dropdown menu suggesting 'Pelican' as a correction.
Domain
Dataset
Hours
Normative
Common Voice (EN) ( Ardila et al., 2020 )
∼ 3,800
Normative
LibriSpeech (EN) ( Panayotov et al., 2015 )
∼ 1,000
Dysarthric
SAP (EN) ( Hasegawa-Johnson et al., 2024 )
∼ 415
Dysarthric
UA-Speech (EN) ( Kim et al., 2008 )
∼ 9
Dysarthric
BF-Sprache (DE) ( Pokel et al., 2025a )
∼ 2.5
Dysarthric
TORGO (EN) ( Rudzicz et al., 2012 )
∼ 3
Table 1. Comparison of hours of existing speech datasets (recording hours are reduced by redundant microphones).
Figure 2. The Adapt4Me Personalization Lifecycle. Step 1 (Speech Recording): The system creates a "Cold Start" profile using a minimized set of 500 words selected via Greedy Biphone Coverage and LLM Prompting ( Pokel et al., 2025a ) . Step 2 (Model Personalization): The backend adapts the model. It uses Semantic Re-chaining (SRC) ( Pokel et al., 2025a ) to generate context, calculates the Phoneme Difficulty Score (PhDScore) to search for highly informative training-samples ( Pokel et al., 2025b ) , and fine-tunes the VI-LoRA adapters ( Pokel et al., 2025c ) (Top Right). Step 3 (Active Learning): The continuous interaction loop. The model transcribes free speech, highlights uncertain words (e.g., “Pylikan”), and generates context-aware N-best suggestions to minimize annotation effort.
Figure 3. Longitudinal user study with a teenage child with structural speech impairment. The system reduces word error rate (WER) from a non-functional ∼ 70% to usable ∼ 25% within under 90 minutes of total interaction. Uncertainty-aware active learning (orange/grey bars) eventually outperforms the full fine-tuning baseline (blue dashed line), demonstrating the effectiveness of VI-LoRA in data-scarce regimes.
Personalizing automatic speech recognition (ASR) systems for non-normative speech, such as dysarthric and aphasic speech, is challenging. While speaker-specific fine-tuning (SS-FT) is widely used, it is typically initialized directly from a generic pre-trained model. Whether speaker-independent adaptation provides a stronger initialization prior under such mismatch remains unclear. In this work, we propose a two-stage adaptation framework consisting of speaker-independent fine-tuning (SI-FT) on multi-speaker non-normative data followed by SS-FT, and evaluate it through a controlled comparison with direct SS-FT under identical per-speaker conditions. Experiments on AphasiaBank and UA-Speech with Whisper-Large-v3 and Qwen3-ASR, alongside evaluation on typical-speech datasets TED-LIUM v3 and FLEURS, show that two-stage adaptation consistently improves personalization while maintaining manageable out-of-domain (OOD) trade-offs.
Shan Jiang, Jiawen Qi, Chuanbing Huo +2
Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands · Penobscot Community Health Care, United States · Department of Computational Linguistics, University of Zurich, Switzerland
Automatic speech recognition (ASR) systems often perform poorly in dysarthric speech, limiting their usefulness to affected speakers in everyday communication. This paper presents a personalized ASR system for a dysarthric speaker, built by adapting a foundation ASR model to speaker-specific data. Using the TEQST tool, we collected 92 hours of read speech and later added 8.8 hours of user corrections gathered through a deployed mobile application. Starting from Whisper, fine-tuning reduced word error rate to 15.8% with only 1.4 hours of adaptation data, reached 10.7% with 22.5 hours, and achieved the best result of 9.7% when using all available data including the corrections. Using LoRA adaptation and/or Qwen3-ASR as foundation model performed worse in this setting. The results show that personalized fine-tuning can make foundation ASR models substantially more effective for dysarthric speech and suitable for practical deployment.
Christian Huber, Laura Kernahan, Alexander Waibel
Interactive Systems Lab Karlsruhe Institute of Technology Karlsruhe, Germany · Interactive Systems Lab Carnegie Mellon University Pittsburgh PA, USA
Automatic Speech recognition (ASR) struggles with Pathological speech due to persistent speaker-specific mismatch. In clinical and accessibility scenarios, only limited target-speaker data is typically available, making full model retraining impractical. We present a component-level analysis of LoRA adaptation in Whisper, investigating how adapter placement interacts with the model size, training efficiency and utterance-duration in single-speaker settings. Our experimental comparative study shows that while scaling reduces errors, it does not eliminate mismatch. Furthermore, the full attention LoRa provides the most reliable accuracy, and the best time--accuracy trade-off. Improvements vary across utterance length, with medium-length recordings benefiting the most, while short and long utterances remain challenging. These findings highlight the importance of adaptation placement and duration-dependent evaluation, under challenging, data-scarce personalization scenarios.
Giuseppe Giaimis, Sergei Katkov, Muhammad Azfar Yaqub +1