Accurate assessment of dysarthric speech is vital for clinical diagnosis and rehabilitation, yet existing automated methods rely heavily on isolated, single-task models or generic speech quality metrics that fail to disentangle heterogeneous neuromotor deficits. This work presents the first systematic investigation into adapting auditory large language models (ALLMs) for multi-dimensional dysarthric speech assessment under a unified, prompt-conditioned framework. Four representative ALLMs are benchmarked across five clinical perceptual dimensions from the Speech Accessibility Project (SAP): naturalness, monopitch, intelligibility, imprecise consonants, and inappropriate silences. To bridge generic perceptual quality and clinical evaluation, a two-stage training paradigm is explored, leveraging out-of-domain pre-training followed by in-domain clinical adaptation. Experimental results demonstrate that leading ALLMs reliably capture fine-grained pathological variations, exhibiting distinct and complementary strengths across prosodic and articulatory attributes. These findings establish ALLMs as a scalable foundation for clinical speech evaluation and provide practical insights into model selection and multi-task transfer potential.
Figures & tables
Figure 1: Distribution of utterances across severity levels (1–7) in the Speech Accessibility Project (SAP) dataset for the five perceptual dimensions: (a) Naturalness, (b) Monopitch, (c) Intelligibility, (d) Imprecise Consonants, and (e) Inappropriate Silences. Higher scores indicate more severe impairment. Notably, several pathological attributes (particularly Inappropriate Silences and Intelligibility) exhibit significant class imbalance toward lower severity levels.
Dimension
Qwen2-Audio
Qwen2.5-Omni
video-SALMONN 2 plus
SALMONN 2
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
Naturalness
0.655
0.684
0.764
0.828
0.836
0.439
0.702
0.738
0.703
0.584
0.644
0.870
Monopitch
0.545
0.534
0.627
0.584
0.595
0.643
0.572
0.567
0.658
0.386
0.406
0.732
Intelligibility
0.549
0.548
0.366
0.610
0.596
0.350
0.635
0.691
0.322
0.592
0.661
0.338
Imprecise Consonants
0.632
0.691
0.449
0.700
0.673
0.411
0.738
0.789
0.342
0.640
0.679
0.456
Inappropriate Silences
0.335
0.371
0.193
0.631
0.750
0.142
0.555
0.616
0.099
0.226
0.298
0.136
Table 1: Comparison of Auditory Large Language Models (ALLMs) on speech quality prediction over the SAP test set. Spearman ( ρ ) and Pearson ( r ) correlations and Mean Absolute Error (MAE) are computed between model predictions and human ratings on five dimensions: naturalness, monopitch, intelligibility, imprecise consonants, and inappropriate silences. Higher ρ and r and lower MAE indicate better performance. Best result per dimension and metric is in bold.
Dimension
Zero-shot
Post-QualiSpeech
Post-SAP
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
Naturalness
0.307
0.327
2.292
0.444
0.437
1.761
0.828
0.836
0.439
Monopitch
0.264
0.267
1.490
0.189
0.190
1.857
0.584
0.595
0.643
Intelligibility
0.328
0.363
1.934
0.150
0.100
2.789
0.610
0.596
0.350
Imprecise Consonants
0.492
0.547
1.714
0.419
0.463
2.215
0.700
0.673
0.411
Inappropriate Silences
− 0.073
− 0.034
3.303
0.098
0.093
3.275
0.631
0.750
0.142
Table 2: Effect of post-training on Qwen2.5-Omni for speech quality prediction over the SAP test set: zero-shot performance (Zero-shot), fine-tuned with QualiSpeech (Post-QualiSpeech) and post-training with SAP (Post-SAP).
Figure 2: Scatter plots of model predictions against human ratings on the SAP test set for (a) Naturalness, (b) Monopitch, (c) Intelligibility, (d) Imprecise Consonants, (e) Inappropriate Silences. These plots visualize the utterance-level predictions corresponding to the Qwen2.5-Omni results reported in Table 1 . Each point represents one test utterance, and the dashed diagonal line indicates perfect agreement between predicted and ground-truth scores.
Dimension
video-SALMONN 2 Plus
Qwen2.5-Omni
Single
Mix
Single
Mix
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
Naturalness
0.722
0.758
0.710
0.702
0.738
0.703
0.848
0.858
0.462
0.828
0.836
0.439
Monopitch
0.510
0.507
0.633
0.572
0.567
0.658
0.693
0.687
0.566
0.584
0.595
0.643
Intelligibility
0.592
0.638
0.346
0.635
0.691
0.322
0.669
0.697
0.311
0.610
0.596
0.350
Imprecise Consonants
0.702
0.758
0.414
0.738
0.789
0.342
0.738
0.736
0.385
0.700
0.673
0.411
Table 3: Systematic comparison of dedicated single-task training (Single) versus unified multi-task training (Mix) across representative Auditory Large Language Models over the SAP test set. video-SALMONN 2 Plus demonstrates substantial multi-task synergy where joint training improves performance across almost all dimensions. Qwen2.5-Omni shows that unified multi-task training achieves competitive accuracy (average ρ within 0.046) with only 1/5 of the parameters, while outperforming single-task models on sparse attributes (Inappropriate Silences) and calibration error (Naturalness MAE). Bold indicates the superior result between Single and Mix for each model.
Automatic speech recognition (ASR) systems remain brittle on dysarthric and other atypical speech. Recent audio-language models raise the possibility of improving performance by conditioning on additional clinical context at inference time, but it is unclear whether these models can make use of such information. We introduce a benchmark built on the Speech Accessibility Project (SAP) dataset that tests whether diagnosis labels, clinician-derived speech ratings, and progressively richer clinical descriptions improve transcription accuracy for dysarthric speech. Across matched comparisons on nine models, we find that current models do not meaningfully use this context: diagnosis-informed and clinically detailed prompts yield negligible improvements and often degrade word error rate. We complement the prompting analysis with context-dependent fine-tuning, showing that LoRA adaptation with a mixture of clinical prompt formats achieves a WER of 0.066, a 52% relative reduction over the frozen baseline, while preserving performance when context is unavailable. Subgroup analyses reveal significant gains for Down syndrome and mild-severity speakers. These results clarify where current models fall short and provide a testbed for measuring progress toward more inclusive ASR.
Pehuén Moure, Niclas Pokel, Bilal Bounajma +4
Institute of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland · Department of Mathematics, ETH Zurich, Switzerland · Department of Computational Linguistics, University of Zurich, Switzerland
Dysarthria is a speech disorder marked by reduced intelligibility and communicative effectiveness. Automatic utterance-level assessment of dysarthric speech can support scalable speech monitoring and therapy-related analysis. Yet training such systems is bottlenecked by the scarcity of clinically annotated dysarthric speech. This work proposes to augment dysarthric speech assessment using data from speech synthesis evaluations, specifically human-annotated utterances with Mean Opinion Score (MOS) labels from the QualiSpeech corpus. Experiments show that fine-tuning on speech synthesis assessment data consistently improves performance on both intelligibility and naturalness prediction, while joint training yields gains primarily on naturalness. These results suggest that synthesis artifacts and dysarthric speech share perceptual commonalities, and speech synthesis evaluation corpora offer a practical augmentation source that reduces reliance on scarce clinical annotations.
Kaimeng Jia, Minzhu Tu, Zengrui Jin +2
Tsinghua University · Beijing University of Posts and Telecommunications
Dysarthric speech quality assessment (DSQA) is critical for clinical diagnostics and inclusive speech technologies. However, subjective evaluation is costly and difficult to scale, and the scarcity of labeled data limits robust objective modeling. To address this, we propose a three-stage framework that leverages unlabeled dysarthric speech and large-scale typical speech datasets to scale training. A teacher model first generates pseudo-labels for unlabeled samples, followed by weakly supervised pretraining using a label-aware contrastive learning strategy that exposes the model to diverse speakers and acoustic conditions. The pretrained model is then fine-tuned for the downstream DSQA task. Experiments on five unseen datasets spanning multiple etiologies and languages demonstrate the robustness of our approach. Our Whisper-based baseline significantly outperforms SOTA DSQA predictors such as SpICE, and the full framework achieves an average SRCC of 0.761 across unseen test datasets.
Jaesung Bae, Xiuwen Zheng, Minje Kim +2
University of Illinois Urbana-Champaign, IL, USA · Korea Advanced Institute of Science & Technology, KR