Dysarthric audio-visual speech recognition (DAVSR) enhances performance by integrating complementary visual cues. However, existing research is largely confined to word-level tasks with limited vocabularies, which severely restricts real-world communication for dysarthric speakers. Furthermore, performance remains heavily dependent on supervised fine-tuning with speaker-specific data; significant variations in age, etiology, and severity cause traditional methods to struggle with unseen speakers. To address these domain shifts and the scarcity of continuous multimodal resources, we first introduce a "source speech search-segmentation" pipeline to construct a structured sentence-level corpus from raw CDSD videos. We then propose the use of unsupervised Test-Time Adaptation (TTA) for sentence-level DAVSR using Sequential-level Generalized Entropy Minimization (SGEM). Our evaluations across the state-of-the-art AV-SUPERB and ViSpeR frameworks demonstrate that TTA consistently alleviates Character Error Rate (CER) degradation for unseen pathological speakers. Paired t-tests verify that these improvements are statistically significant (p < 0.05), highlighting TTA as a promising paradigm for advancing speaker-adaptive assistive speech interfaces.
Figures & tables
Figure 1: The workflow of the source speech search-segmentation procedure. The framework processes raw audio-visual streams through three core stages: (1) monotonic segmentation to ensure chronological consistency; (2) Fbank feature extraction and distance calculation to locate candidates via minimum Euclidean distance; and (3) multi-modal manual result verification and refinement to filter out no-match cases and finalize the aligned sentence-level corpus.
ID
Segmented Speech
Raw Video
Aligned
Availability
Count
Range (s)
Count
Range (s)
Data
Rate
01
1107
[0.5, 10.8]
22
[136.2, 279.6]
1107
100.0%
02
988
[0.4, 12.6]
12
[237.6, 348.6]
968
98.0%
03
926
[0.4, 15.7]
12
[154.8, 585.6]
912
98.5%
04
1035
[0.5, 10.4]
14
[252.6, 307.2]
1035
100.0%
06
969
[0.6, 9.2]
9
[297.6, 744.0]
967
99.8%
Table 1: Statistics of the CDSD dataset.
Figure 2: TTA was evaluated on two audio-visual speech recognition frameworks: AV-SUPERB and ViSpeR. Colors indicate different TTA setups, identifying the audio feature extractor, video feature extractor, cross-modal fusion layer, encoder, layer normalization (LayerNorm) layers, and decoder, respectively. Color viewing is recommended.
Figure 3: Qualitative examples of speech recognition transcripts from speakers 06 and 44 under their optimal model (ViSpeR), demonstrating the error-correction capability of the TTA. ΔCER quantifies the absolute reduction in CER.
Figure 4: Distribution of testing utterances per speaker, categorized by the level of CER reduction achieved through TTA across four distinct evaluation frameworks: AS-1, AS-2, AS-3, and ViSpeR. To compensate for visual limitations, the samples are binned into four precise CER improvement intervals ( ΔCER=CERunadapted−CERTTA ): marginal-to-low correction ( 25%>ΔCER≥0% , blue bars), low-to-moderate correction ( 50%>ΔCER≥25% , orange bars), moderate-to-high correction ( 75%>ΔCER≥50% , green bars), and significant correction ( ΔCER≥75% , red bars).
Speaker ID
Model
Baseline
TTA-1
TTA-3
TTA-5
TTA-10
Δ
AS-1
88.4
87.6
87.1
87.8
87.7
1.3
01
AS-2
74.4
72.1
71.0
71.7
72.2
3.4
AS-3
75.0
73.0
72.5
72.7
73.3
2.5
ViSpeR
58.9
55.7
48.1
47.3
47.1*
11.8
AS-1
85.2
85.5
84.3
84.3
84.4
0.9
02
AS-2
88.2
87.6
87.4
87.4
88.1
0.8
Table 2: TTA Performance (CER, %) on unseen speakers across various adaptation iterations. The absolute CER reduction ( Δ ) is computed as the baseline error minus the optimal adapted error ( CERBaseline−CEROptimal ), with the latter denoted by the bolded values in each row. The asterisk ( ∗ ) highlights the best recognition performance achieved per speaker across all evaluated configurations.
Model
Setting
Baseline
TTA-1
TTA-3
TTA-5
TTA-10
Δ
LayerNorm
88.4
87.7
87.6
87.7
88.5
0.8
Feature
88.4
87.6
87.1
87.8
87.7
1.3
AS-1
Encoder
88.4
87.7
87.5
87.9
88.7
0.9
Decoder
88.4
87.8
87.6
87.9
88.7
0.8
All
88.4
87.6
87.6
87.7
88.2
0.8
LayerNorm
74.4
72.4
72.1
73.1
73.2
2.3
Table 3: The performance of different TTA settings for speaker 01 under various frameworks (CER, %).
Dysarthric speech reconstruction (DSR) typically employs a cascaded system that combines automatic speech recognition (ASR) and sentence-level text-to-speech (TTS) to convert dysarthric speech into normally-prosodied speech. However, dysarthric individuals often speak more slowly, leading to excessively long response times in such systems, rendering them impractical in long-speech scenarios. Cascaded DSR systems based on streaming ASR and incremental TTS can help reduce latency. However, patients with differing dysarthria severity exhibit substantial pronunciation variability for the same text, resulting in poor robustness of ASR and limiting the intelligibility of reconstructed speech. In addition, incremental TTS suffers from poor prosodic feature prediction due to a limited receptive field. In this study, we propose an end-to-end simultaneous DSR system with two key innovations: 1) A frame-level adaptor module is introduced to bridge ASR and TTS. By employing explicit-implicit semantic information fusion and joint module training, it enhances the error tolerance of TTS to ASR outputs. 2) A multiple wait-k autoregressive TTS module is designed to mitigate prosodic degradation via multi-view knowledge distillation. Our system has an average response time of 1.03 seconds on Tesla A100, with an average real-time factor (RTF) of 0.71. On the UASpeech dataset, it attains a mean opinion score (MOS) of 4.67 and demonstrates a 54.25% relative reduction in word error rate (WER) compared to the state-of-the-art. Our demo is available at: https://wflrz123.github.io/
Minghui Wu, Haitao Tang, Jiahuan Fan +2
University of Science and Technology of China, China · iFlytek Co., Ltd., China
Dysarthric speech exhibits abnormal prosody and significant speaker variability, presenting persistent challenges for automatic speech recognition (ASR). While text-to-speech (TTS)-based data augmentation has shown potential, existing methods often fail to accurately model the pathological rhythm and acoustic style of dysarthric speech. To address this, we propose DARS, a dysarthria-aware rhythm-style synthesis framework based on the Matcha-TTS architecture. DARS incorporates a multi-stage rhythm predictor optimized by contrastive preferences between normal and dysarthric speech, along with a dysarthric-style conditional flow matching mechanism, jointly enhancing temporal rhythm reconstruction and pathological acoustic style simulation. Experiments on the TORGO dataset demonstrate that DARS achieves a Mean Cepstral Distortion (MCD) of 4.29, closely approximating real dysarthric speech. Adapting a Whisper-based ASR system with synthetic dysarthric speech from DARS achieves a 54.22% relative reduction in word error rate (WER) compared to state-of-the-art methods, demonstrating the framework's effectiveness in enhancing recognition performance.
Minghui Wu, Xueling Liu, Jiahuan Fan +3
University of Science and Technology of China, China · iFlytek Co., Ltd., China · Huawei Technology, China
Automatic speech recognition (ASR) systems often perform poorly in dysarthric speech, limiting their usefulness to affected speakers in everyday communication. This paper presents a personalized ASR system for a dysarthric speaker, built by adapting a foundation ASR model to speaker-specific data. Using the TEQST tool, we collected 92 hours of read speech and later added 8.8 hours of user corrections gathered through a deployed mobile application. Starting from Whisper, fine-tuning reduced word error rate to 15.8% with only 1.4 hours of adaptation data, reached 10.7% with 22.5 hours, and achieved the best result of 9.7% when using all available data including the corrections. Using LoRA adaptation and/or Qwen3-ASR as foundation model performed worse in this setting. The results show that personalized fine-tuning can make foundation ASR models substantially more effective for dysarthric speech and suitable for practical deployment.
Christian Huber, Laura Kernahan, Alexander Waibel
Interactive Systems Lab Karlsruhe Institute of Technology Karlsruhe, Germany · Interactive Systems Lab Carnegie Mellon University Pittsburgh PA, USA