cs.SDNov 8, 2026

DIY Customized Speech Recognition for a Person with Speech Disabilities

Authors: Paulo A. Condado, Fernando G. Lobo

Organizations: CENSE – Center for Environmental and Sustainability Research & CHANGE - Global Change and Sustainability Institute NOVA School of Science and Technology, NOVA University Lisbon Campus de Caparica, 2829-516 Caparica · CENSE – Center for Environmental and Sustainability Research & CHANGE - Global Change and Sustainability Institute DEEI-FCT, Universidade do Algarve Campus de Gambelas, 8005-139 Faro, Portugal

Abstract

General-purpose state-of-the-art automatic speech recognition (ASR) systems fail systematically for speakers with moderate to severe speech disabilities. This paper describes the design, implementation, and evaluation of a customized ASR system built for the first author, a researcher with cerebral palsy and dysarthria. We fine-tuned OpenAI's Whisper medium and large-v3 models using Low-Rank Adaptation (LoRA) on audio samples recorded from the first author's own speech on consumer-grade hardware. On a held-out test set of 100 previously unseen utterances, the corresponding transcriptions resulted in a Word Error Rate drop from 49% to 17% for medium and from 30% to 12% for large-v3, while the Character Error Rate fell from 26% to 7% for medium and from 16% to 5% for large-v3, indicating that the residual errors are minor phonemic substitutions. Based on these models, the first author developed AVoz, a mobile application that acts as a real-time interpreter, converting his speech into synthesized voice for the listener. All processing is performed locally, ensuring that no audio data leaves the user's machine. We argue that the methodology is replicable. Moreover, we publish the full LoRA configuration, the implementation, and a step-by-step recording and training protocol, so that other people with speech disabilities, or their family members and caregivers, can build their own systems with a few hours of labeled audio data and a couple of days of GPU time.

Figures & tables

Explore similar work

CardsList
  1. Adapting Foundation ASR Models to Dysarthric Speech: A Case Study

    Jun 30, 2026Christian Huber, Laura Kernahan, Alexander WaibelPersonalized Speech RecognitionAutomatic Speech Recognition

  2. Personalized Automatic Speech Recognition for a Dysarthric and Tracheostomic Speaker using Artificial Conversations

    Oct 2, 2026David Nadrchal, Monorama Swain, Florian Schmid +2Personalized Speech RecognitionAutomatic Speech Recognition