Dysarthric speech reconstruction (DSR) typically employs a cascaded system that combines automatic speech recognition (ASR) and sentence-level text-to-speech (TTS) to convert dysarthric speech into normally-prosodied speech. However, dysarthric individuals often speak more slowly, leading to excessively long response times in such systems, rendering them impractical in long-speech scenarios. Cascaded DSR systems based on streaming ASR and incremental TTS can help reduce latency. However, patients with differing dysarthria severity exhibit substantial pronunciation variability for the same text, resulting in poor robustness of ASR and limiting the intelligibility of reconstructed speech. In addition, incremental TTS suffers from poor prosodic feature prediction due to a limited receptive field. In this study, we propose an end-to-end simultaneous DSR system with two key innovations: 1) A frame-level adaptor module is introduced to bridge ASR and TTS. By employing explicit-implicit semantic information fusion and joint module training, it enhances the error tolerance of TTS to ASR outputs. 2) A multiple wait-k autoregressive TTS module is designed to mitigate prosodic degradation via multi-view knowledge distillation. Our system has an average response time of 1.03 seconds on Tesla A100, with an average real-time factor (RTF) of 0.71. On the UASpeech dataset, it attains a mean opinion score (MOS) of 4.67 and demonstrates a 54.25% relative reduction in word error rate (WER) compared to the state-of-the-art. Our demo is available at: https://wflrz123.github.io/
Figures & tables
Figure 1: Cascaded DSR system based on ASR and TTS.
Figure 2: End-to-end simultaneous DSR system.
ID
Model
3-rd Level
4-th Level
WER
MOS
WER
MOS
E1
Original
49.42
2.08
33.71
2.48
E2
ASR+sentence-level TTS
25.93
3.32
12.91
4.71
E3
Adaptor w/ Eq. (5)
25.63
3.27
12.89
4.24
E4
Adaptor w/ Eq. (6)
25.07
3.41
12.23
4.53
E5
Adaptor w/ Eq. (5), Eq. (6)
23.67
3.64
11.55
4.76
Table 1: Results of the adaptor module on commercial dataset.
ID
Model
3-rd Level
4-th Level
WER
MOS
WER
MOS
E1
Original
49.42
2.08
33.71
2.48
E7
wait-k = 1
24.83
3.28
14.17
4.02
E6
wait-k = 10
22.58
3.66
10.84
4.79
E8
wait-k = 20
21.07
4.44
9.73
4.84
E9
wait-k = 1 w/ MTL
24.61
3.30
14.08
4.05
Table 2: Results of the multi-task wait-k on incremental TTS.
Figure 3: Comparison prosody on different receptive field for incremental TTS . (a) is E7; (b) is E8.
ID
Params (M)
FLOPS (M)
System Response Time (second)
RTF
E2
248.92
400.48
2.67
1.39
E7
232.83
350.15
0.61
0.70
E6
1.03
0.71
E8
1.38
0.71
Table 3: Performance of different DSR Systems.
ID
Model
Metric
M05 (mid)
F04 (mid)
M07 (low)
F02 (low)
E11
Original
WER
81.70
81.70
95.60
95.90
MOS
2.86
2.42
1.75
1.95
E12
E2E-DSR
WER
69.80
69.30
73.10
72.00
MOS
3.61
3.47
3.72
3.87
E13
ASA-DSR
WER
62.50
76.60
62.70
65.80
MOS
4.19
3.98
3.71
4.26
Table 4: The 5-scale MOS test scores and WER (%) comparison E2E-SDSR and SOTA for UASpeech.
Dysarthric speech exhibits abnormal prosody and significant speaker variability, presenting persistent challenges for automatic speech recognition (ASR). While text-to-speech (TTS)-based data augmentation has shown potential, existing methods often fail to accurately model the pathological rhythm and acoustic style of dysarthric speech. To address this, we propose DARS, a dysarthria-aware rhythm-style synthesis framework based on the Matcha-TTS architecture. DARS incorporates a multi-stage rhythm predictor optimized by contrastive preferences between normal and dysarthric speech, along with a dysarthric-style conditional flow matching mechanism, jointly enhancing temporal rhythm reconstruction and pathological acoustic style simulation. Experiments on the TORGO dataset demonstrate that DARS achieves a Mean Cepstral Distortion (MCD) of 4.29, closely approximating real dysarthric speech. Adapting a Whisper-based ASR system with synthetic dysarthric speech from DARS achieves a 54.22% relative reduction in word error rate (WER) compared to state-of-the-art methods, demonstrating the framework's effectiveness in enhancing recognition performance.
Minghui Wu, Xueling Liu, Jiahuan Fan +3
University of Science and Technology of China, China · iFlytek Co., Ltd., China · Huawei Technology, China
In our goal to develop personalised dysarthric speech recognition (DSR) models, this study compared the recognition performances of human listeners and those of three state-of-the-art, off-the-shelf ASR systems (Whisper-large-V3, Google Chirp 3, and Omnilingual) on the recognition of Dutch continuous read and spontaneous speech from a single speaker with severe dysarthria. Results showed that both humans listeners and the three off-the-shelf ASR systems exhibit word error rates (WER) exceeding 70% on average, indicating that DSR is highly challenging for both humans and ASR systems. Fine-tuning on the dysarthric speech significantly reduced WER. Although overall WERs are still quite high (>23%), the personalised DSR models outperformed the human listeners, and performance is getting closer to being useful for supporting day-to-day communication of dysarthric speakers. Future research should focus on improving personalized DSR on spontaneous speech and longer utterances in the case of read speech, with a specific focus on particular phonemes.
Yuanyuan Zhang, Dimme de Groot, Jorge Martinez +1
Multimedia Computing Group Delft University of Technology Delft, the Netherlands
Dysarthric audio-visual speech recognition (DAVSR) enhances performance by integrating complementary visual cues. However, existing research is largely confined to word-level tasks with limited vocabularies, which severely restricts real-world communication for dysarthric speakers. Furthermore, performance remains heavily dependent on supervised fine-tuning with speaker-specific data; significant variations in age, etiology, and severity cause traditional methods to struggle with unseen speakers. To address these domain shifts and the scarcity of continuous multimodal resources, we first introduce a "source speech search-segmentation" pipeline to construct a structured sentence-level corpus from raw CDSD videos. We then propose the use of unsupervised Test-Time Adaptation (TTA) for sentence-level DAVSR using Sequential-level Generalized Entropy Minimization (SGEM). Our evaluations across the state-of-the-art AV-SUPERB and ViSpeR frameworks demonstrate that TTA consistently alleviates Character Error Rate (CER) degradation for unseen pathological speakers. Paired t-tests verify that these improvements are statistically significant (p < 0.05), highlighting TTA as a promising paradigm for advancing speaker-adaptive assistive speech interfaces.
Shiyao Wang, Hui Wang, Jiaming Zhou +2
Academy for Advanced Interdisciplinary Studies, Nankai University, Tianjin, China