Dysarthric speech quality assessment (DSQA) is critical for clinical diagnostics and inclusive speech technologies. However, subjective evaluation is costly and difficult to scale, and the scarcity of labeled data limits robust objective modeling. To address this, we propose a three-stage framework that leverages unlabeled dysarthric speech and large-scale typical speech datasets to scale training. A teacher model first generates pseudo-labels for unlabeled samples, followed by weakly supervised pretraining using a label-aware contrastive learning strategy that exposes the model to diverse speakers and acoustic conditions. The pretrained model is then fine-tuned for the downstream DSQA task. Experiments on five unseen datasets spanning multiple etiologies and languages demonstrate the robustness of our approach. Our Whisper-based baseline significantly outperforms SOTA DSQA predictors such as SpICE, and the full framework achieves an average SRCC of 0.761 across unseen test datasets.
Figures & tables
Lang.
Name
Etiology
LBL
Hours
# Spk
Training Data
EN
SAP [ 6 ] Train
ALS, CP, DS, PD
✓
10.8
318
✗
232.9
722
EN
LibriSpeech [ 10 ]
Healthy
–
921.7
2.3k
In-domain Evaluation Data
EN
SAP [ 6 ] Test
ALS, CP, DS, PD
✓
3.1
89
Table 1: Summary of dataset information. Training is English-only, while cross-domain tests are multi-lingual. EN: English; ZH: Mandarin Chinese; IT: Italian; CS: Czech; SK: Slovak; ES: Spanish. ALS: Amyotrophic lateral sclerosis; CP: Cerebral palsy; DS: Down syndrome; PD: Parkinson’s disease; AD: Alzheimer’s disease; MCI: Mild Cognitive Impairment. DysArinVox provides fine-grained annotations across 20 disorder categories (e.g., paralysis, leukoplakia, and polyps), and EasyCall includes dysarthria associated with PD, Huntington’s disease (HD), ALS, peripheral neuropathy, and myopathic or myasthenic lesions. LBL and Spk indicate the labeling method and the number of speakers, respectively.
Figure 1: (a–c) Illustration of the three-stage framework with weakly supervised pretraining, and (d) the proposed pairing strategies for weakly supervised contrastive learning. (a) Stage 1: A regression model is trained on the labeled SAP dataset (3% of the total), and pseudo-labels are generated for the unlabeled portion. (b) Stage 2: Three linear layers are trained with weakly supervised contrastive losses. Pseudo-labeled SAP data from Stage 1 and LibriSpeech are additionally used, with LibriSpeech assigned a label of one (healthy). (c) Stage 3: The first two linear layers from Stage 2 are fine-tuned with an additional final linear layer using the labeled SAP dataset. (d) In Stage 2 , positive pairs for contrastive learning are generated in three ways: discretizing the label, thresholding the distance between continuous labels, or dividing the label range into a binary dichotomy. With anchor y~i=2.4 , samples with y~j in the blue region form positive pairs: for example, y~j=2.6 is negative under discrete pairing but positive under continuous pairing. Binary pairing divides the data into two groups, providing weaker supervision.
Figure 2: Histogram of label proportions in Dlabeled (left) and Dpseudo (right).
Model
Stage
In-domain
Cross-domain
SAP
UASpeech
DysArinVox
EasyCall
EWA-DB
NeuroVoz
Average
1
2
3
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
DNSMOS [ 1 ]
0.186
0.278
0.750
0.742
0.370
0.502
0.041
0.061
-0.274
-0.294
-0.361
-0.363
0.105
0.130
UTMOS [ 2 ]
0.489
0.499
0.962
0.912
0.521
0.525
-0.051
-0.108
0.328
0.345
-0.028
-0.065
0.346
0.322
SpICE [ 3 ]
0.473
0.505
0.936
0.926
0.538
0.463
0.205
0.237
0.393
0.366
0.180
0.201
0.450
0.439
HuBERT Probe [ 5 ]
0.531
0.620
0.927
0.924
0.279
0.334
0.604
0.704
0.588
0.487
0.705
0.706
0.621
0.631
Table 2: Performance comparison on in-domain and cross-domain test sets. “Average” in cross-domain indicates the average performance across all cross-domain datasets. Bold and underlined values denote the best and second-best results. Across five seeds, the standard deviations of the in-domain and average cross-domain SRCCs for Proposed ( Lcoarse ) are 0.0021 and 0.0041, respectively.
Figure 3: t-SNE figures after stage 2 with various contrastive loss choices. We randomly select 1000 samples in LibriSpeech and SAP training data. These are the embeddings right after the temporal pooling. Blue cross represents the LibriSpeech data, and circles indicate the SAP dataset. From red to green, the color indicates the low-severity to high-severity levels. Best viewed in color.
Index
Stage
In-domain
Cross-domain
SAP
UASpeech
DysArinVox
EasyCall
EWA-DB
NeuroVoz
Average
1
2
3
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
1
✓
0.719
0.755
0.949
0.963
0.578
0.619
0.849
0.783
0.709
0.686
0.575
0.601
0.732
0.730
2
✓
✓
0.719
0.755
0.949
0.963
0.577
0.620
0.856
0.785
0.709
0.686
0.571
0.598
0.732
0.730
3
✓
✓
0.681
0.740
0.960
0.968
0.604
0.654
0.902
0.850
0.670
0.667
0.596
0.621
0.746
0.752
4
✓
✓
✓
0.716
0.750
0.975
0.976
0.631
0.632
0.872
0.810
0.711
0.695
0.617
0.630
0.761
0.749
Table 3: Ablation study of each stage of the Proposed ( Lcoarse ) model.
Model
In-domain
Cross-domain
SAP
UASpeech
DysArinVox
EasyCall
EWA-DB
NeuroVoz
Average
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
SRCC
PCC
Proposed ( Lcoarse )
0.716
0.750
0.975
0.976
0.631
0.632
0.872
0.810
0.711
0.695
0.617
0.630
0.761
0.749
w/o DLibri
0.706
0.743
0.952
0.964
0.605
0.630
0.832
0.746
0.692
0.668
0.590
0.613
0.734
0.724
w/o Dunlabeled
0.715
0.755
0.950
0.956
0.609
0.625
0.877
0.799
0.681
0.660
0.632
0.659
0.750
0.740
w/o Lvar
0.711
0.753
0.963
0.978
0.661
0.658
0.862
0.813
0.683
0.660
0.606
0.616
0.755
0.745
Table 4: Ablation study on additional LibriSpeech dataset, unlabeled portion of SAP dataset, and variance regularization objective.
Figure 4: The improvement percentages of SRCC and PCC over the Baseline model vary with different values of τ . (a) In-domain testset (SAP dataset) and (b) average scores of cross-domain testsets. In general, the performance of our proposed methods improves as τ increases. Although SimCLR achieves the best cross-domain average performance at τ=10 , its in-domain test performance deteriorates significantly, highlighting the robustness of our proposed methods.
Figure 5: Embedding spaces with different τ . Since the LibriSpeech and SAP datasets have distinct characteristics, they can be considered easy positive/negative pairs. Therefore, with a small τ , contrastive learning tends not to associate pairs from the LibriSpeech and SAP datasets. In contrast, with a large τ , they are better harmonized, enhancing the robustness of the downstream regression model.
Dysarthria is a speech disorder marked by reduced intelligibility and communicative effectiveness. Automatic utterance-level assessment of dysarthric speech can support scalable speech monitoring and therapy-related analysis. Yet training such systems is bottlenecked by the scarcity of clinically annotated dysarthric speech. This work proposes to augment dysarthric speech assessment using data from speech synthesis evaluations, specifically human-annotated utterances with Mean Opinion Score (MOS) labels from the QualiSpeech corpus. Experiments show that fine-tuning on speech synthesis assessment data consistently improves performance on both intelligibility and naturalness prediction, while joint training yields gains primarily on naturalness. These results suggest that synthesis artifacts and dysarthric speech share perceptual commonalities, and speech synthesis evaluation corpora offer a practical augmentation source that reduces reliance on scarce clinical annotations.
Kaimeng Jia, Minzhu Tu, Zengrui Jin +2
Tsinghua University · Beijing University of Posts and Telecommunications
Dysarthric speech recognition is crucial for facilitating effective communication among individuals with dysarthria. However, accurately recognizing dysarthric speech poses significant challenges due to varying severity levels and limited data availability. In this paper, we explore data augmentation techniques for dysarthric automatic speech recognition (ASR) systems by fine-tuning the End-to-End pre-trained Wav2Vec2 model, with a specific focus on severity levels. To address the challenges of data scarcity and the need for extensive data in fine-tuning pre-trained ASR systems for dysarthric speech, we investigate four prominent data augmentation methods: Speaking-Rate Modification (SRM), Pitch Modification (PM), Formant Modification (FM), and vocal tract Length Perturbation (VTLP), tailored to different aspects of dysarthria. The study uses individually fine-tuned Wav2Vec2 models for each severity class as baseline systems. Additionally, we conducted severity-specific fine-tuning of the ASR model using augmented data. Results demonstrate distinct efficacy patterns for each augmentation technique across severity levels. The best WERs were achieved with SRM (s=0.8) for low (9.02%) and medium (38.11%) severities, and with PM (τ=0.8) for high severity (55.15%), reflecting relative improvements of 30.02%, 16.64%, and 15.47%, respectively. These results confirm the effectiveness of the augmentation methods in improving dysarthric ASR performance.
Department of Electronics and Communication Engineering, National Institute of Technology Sikkim, India · Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California, Los Angeles, USA
Automatic dysarthria severity assessment is limited by the scarcity of labeled pathological speech data. To address this, we propose Cross-lingual Retrieval-Augmented Classification (CRAC), which leverages speech from a different language via an align-retrieve-fuse pipeline. Supervised contrastive learning first shapes a severity-focused embedding space, then a vector database is built from the opposite-language corpus. During both training and inference, the classifier retrieves top-k references from the aligned space and fuses them with the input via cross-attention. Evaluated on Korean post-stroke and Italian ALS dysarthria datasets under a speaker-independent three-class protocol, CRAC achieves balanced accuracies of 87.3% on Korean and 86.7% on Italian, improving over monolingual baselines by 8.4 and 20.0 percentage points, respectively.
Taeyoung Jeong, Insung Lee, Du-Seong Chang +1
Department of Artificial Intelligence, Sogang University, South Korea