Something from Nothing: Data Augmentation for Robust Severity Level Estimation of Dysarthric Speech

Jaesung Bae; Xiuwen Zheng; Minje Kim; Chang D. Yoo; Mark Hasegawa-Johnson

eess.ASMar 16, 2026

Something from Nothing: Data Augmentation for Robust Severity Level Estimation of Dysarthric Speech

Authors: Jaesung Bae, Xiuwen Zheng, Minje Kim, Chang D. Yoo, Mark Hasegawa-Johnson

Organizations: University of Illinois Urbana-Champaign, IL, USA · Korea Advanced Institute of Science & Technology, KR

Abstract

Dysarthric speech quality assessment (DSQA) is critical for clinical diagnostics and inclusive speech technologies. However, subjective evaluation is costly and difficult to scale, and the scarcity of labeled data limits robust objective modeling. To address this, we propose a three-stage framework that leverages unlabeled dysarthric speech and large-scale typical speech datasets to scale training. A teacher model first generates pseudo-labels for unlabeled samples, followed by weakly supervised pretraining using a label-aware contrastive learning strategy that exposes the model to diverse speakers and acoustic conditions. The pretrained model is then fine-tuned for the downstream DSQA task. Experiments on five unseen datasets spanning multiple etiologies and languages demonstrate the robustness of our approach. Our Whisper-based baseline significantly outperforms SOTA DSQA predictors such as SpICE, and the full framework achieves an average SRCC of 0.761 across unseen test datasets.

Figures & tables

Lang.	Name	Etiology	LBL	Hours	# Spk
Training Data
EN	SAP [ 6 ] Train	ALS, CP, DS, PD	✓	10.8	318
EN	SAP [ 6 ] Train	ALS, CP, DS, PD	✗	232.9	722
EN	LibriSpeech [ 10 ]	Healthy	–	921.7	2.3k
In-domain Evaluation Data
EN	SAP [ 6 ] Test	ALS, CP, DS, PD	✓	3.1	89

Table 1: Summary of dataset information. Training is English-only, while cross-domain tests are multi-lingual. EN: English; ZH: Mandarin Chinese; IT: Italian; CS: Czech; SK: Slovak; ES: Spanish. ALS: Amyotrophic lateral sclerosis; CP: Cerebral palsy; DS: Down syndrome; PD: Parkinson’s disease; AD: Alzheimer’s disease; MCI: Mild Cognitive Impairment. DysArinVox provides fine-grained annotations across 20 disorder categories (e.g., paralysis, leukoplakia, and polyps), and EasyCall includes dysarthria associated with PD, Huntington’s disease (HD), ALS, peripheral neuropathy, and myopathic or myasthenic lesions. LBL and Spk indicate the labeling method and the number of speakers, respectively.

Figure 1: (a–c) Illustration of the three-stage framework with weakly supervised pretraining, and (d) the proposed pairing strategies for weakly supervised contrastive learning. (a) Stage 1: A regression model is trained on the labeled SAP dataset (3% of the total), and pseudo-labels are generated for the unlabeled portion. (b) Stage 2: Three linear layers are trained with weakly supervised contrastive losses. Pseudo-labeled SAP data from Stage 1 and LibriSpeech are additionally used, with LibriSpeech assigned a label of one (healthy). (c) Stage 3: The first two linear layers from Stage 2 are fine-tuned with an additional final linear layer using the labeled SAP dataset. (d) In Stage 2 , positive pairs for contrastive learning are generated in three ways: discretizing the label, thresholding the distance between continuous labels, or dividing the label range into a binary dichotomy. With anchor

\tilde{y}_{i}=2.4

, samples with

\tilde{y}_{j}

in the blue region form positive pairs: for example,

\tilde{y}_{j}=2.6

is negative under discrete pairing but positive under continuous pairing. Binary pairing divides the data into two groups, providing weaker supervision.

Panel of Figure 2 — Figure 2: Histogram of label proportions in $\mathcal{D}_{\text{labeled}}$ (left) and $\mathcal{D}_{\text{pseudo}}$ (right).

Model	Stage			In-domain		Cross-domain
	Stage			SAP		UASpeech		DysArinVox		EasyCall		EWA-DB		NeuroVoz		Average
	1	2	3	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC
DNSMOS [ 1 ]				0.186	0.278	0.750	0.742	0.370	0.502	0.041	0.061	-0.274	-0.294	-0.361	-0.363	0.105	0.130
UTMOS [ 2 ]				0.489	0.499	0.962	0.912	0.521	0.525	-0.051	-0.108	0.328	0.345	-0.028	-0.065	0.346	0.322
SpICE [ 3 ]				0.473	0.505	0.936	0.926	0.538	0.463	0.205	0.237	0.393	0.366	0.180	0.201	0.450	0.439
HuBERT Probe [ 5 ]				0.531	0.620	0.927	0.924	0.279	0.334	0.604	0.704	0.588	0.487	0.705	0.706	0.621	0.631

Table 2: Performance comparison on in-domain and cross-domain test sets. “Average” in cross-domain indicates the average performance across all cross-domain datasets. Bold and underlined values denote the best and second-best results. Across five seeds, the standard deviations of the in-domain and average cross-domain SRCCs for Proposed (

\mathcal{L}_{\text{coarse}}

) are 0.0021 and 0.0041, respectively.

Panel of Figure 3 — Figure 3: t-SNE figures after stage 2 with various contrastive loss choices. We randomly select 1000 samples in LibriSpeech and SAP training data. These are the embeddings right after the temporal pooling. Blue cross represents the LibriSpeech data, and circles indicate the SAP dataset. From red to green, the color indicates the low-severity to high-severity levels. Best viewed in color.

Index	Stage			In-domain		Cross-domain
	Stage			SAP		UASpeech		DysArinVox		EasyCall		EWA-DB		NeuroVoz		Average
	1	2	3	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC
1			✓	0.719	0.755	0.949	0.963	0.578	0.619	0.849	0.783	0.709	0.686	0.575	0.601	0.732	0.730
2	✓		✓	0.719	0.755	0.949	0.963	0.577	0.620	0.856	0.785	0.709	0.686	0.571	0.598	0.732	0.730
3		✓	✓	0.681	0.740	0.960	0.968	0.604	0.654	0.902	0.850	0.670	0.667	0.596	0.621	0.746	0.752
4	✓	✓	✓	0.716	0.750	0.975	0.976	0.631	0.632	0.872	0.810	0.711	0.695	0.617	0.630	0.761	0.749

Table 3: Ablation study of each stage of the Proposed (

\mathcal{L}_{\text{coarse}}

) model.

Model	In-domain		Cross-domain
	SAP		UASpeech		DysArinVox		EasyCall		EWA-DB		NeuroVoz		Average
	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC	SRCC	PCC
Proposed ( $\mathcal{L}_{\text{coarse}}$ )	0.716	0.750	0.975	0.976	0.631	0.632	0.872	0.810	0.711	0.695	0.617	0.630	0.761	0.749
w/o $\mathcal{D}_{\text{Libri}}$	0.706	0.743	0.952	0.964	0.605	0.630	0.832	0.746	0.692	0.668	0.590	0.613	0.734	0.724
w/o $\mathcal{D}_{\text{unlabeled}}$	0.715	0.755	0.950	0.956	0.609	0.625	0.877	0.799	0.681	0.660	0.632	0.659	0.750	0.740
w/o $\mathcal{L}_{\text{var}}$	0.711	0.753	0.963	0.978	0.661	0.658	0.862	0.813	0.683	0.660	0.606	0.616	0.755	0.745

Table 4: Ablation study on additional LibriSpeech dataset, unlabeled portion of SAP dataset, and variance regularization objective.

Panel of Figure 4 — Figure 4: The improvement percentages of SRCC and PCC over the Baseline model vary with different values of $\tau$ . (a) In-domain testset (SAP dataset) and (b) average scores of cross-domain testsets. In general, the performance of our proposed methods improves as $\tau$ increases. Although SimCLR achieves the best cross-domain average performance at $\tau=10$ , its in-domain test performance deteriorates significantly, highlighting the robustness of our proposed methods.

Panel of Figure 5 — Figure 5: Embedding spaces with different $\tau$ . Since the LibriSpeech and SAP datasets have distinct characteristics, they can be considered easy positive/negative pairs. Therefore, with a small $\tau$ , contrastive learning tends not to associate pairs from the LibriSpeech and SAP datasets. In contrast, with a large $\tau$ , they are better harmonized, enhancing the robustness of the downstream regression model.

Speech Quality Assessment

Speech quality assessment measures how well speech preserves perceptual quality, intelligibility, pronunciation, naturalness, and speaker characteristics.

See all 69 papers

Augmenting Dysarthric Speech Severity Assessment with MOS Supervision
Jun 17, 2026Kaimeng Jia, Minzhu Tu, Zengrui Jin +2Speech Quality Assessment Speech Processing
Tsinghua University · Beijing University of Posts and Telecommunications
Improving End-to-End Speech Recognition for Dysarthric Speech through In-Domain Data Augmentation
Jun 18, 2026Paban Sapkota, Hemant Kumar Kathania, Sudarsana Reddy Kadiri +1Data Augmentation Automatic Speech Recognition
Department of Electronics and Communication Engineering, National Institute of Technology Sikkim, India · Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California, Los Angeles, USA
Cross-lingual Retrieval-Augmented Classification for Dysarthria Severity Assessment
Jun 22, 2026Taeyoung Jeong, Insung Lee, Du-Seong Chang +1Retrieval-Augmented Classification Speech Quality Assessment
Department of Artificial Intelligence, Sogang University, South Korea

Jun 17, 2026J/K move · Enter open · S save

Augmenting Dysarthric Speech Severity Assessment with MOS Supervision

Kaimeng Jia, Minzhu Tu, Zengrui Jin, Siyin Wang+1

Tsinghua University · Beijing University of Posts and Telecommunications

Speech Quality Assessment Speech Processing Dysarthria Assessment

Open paper AskPDF arXiv

Dysarthria is a speech disorder marked by reduced intelligibility and communicative effectiveness. Automatic utterance-level assessment of dysarthric speech can support scalable speech monitoring and therapy-related analysis. Yet training such systems is bottlenecked by the scarcity of clinically annotated dysarthric speech. This work proposes to augment dysarthric speech assessment using data from speech synthesis evaluations, specifically human-annotated utterances with Mean Opinion Score (MOS) labels from the QualiSpeech corpus. Experiments show that fine-tuning on speech synthesis assessment data consistently improves performance on both intelligibility and naturalness prediction, while joint training yields gains primarily on naturalness. These results suggest that synthesis artifacts and dysarthric speech share perceptual commonalities, and speech synthesis evaluation corpora offer a practical augmentation source that reduces reliance on scarce clinical annotations.

Abstract

Speech Quality Assessment

Explore similar work

Augmenting Dysarthric Speech Severity Assessment with MOS Supervision

Improving End-to-End Speech Recognition for Dysarthric Speech through In-Domain Data Augmentation

Cross-lingual Retrieval-Augmented Classification for Dysarthria Severity Assessment