Accurate assessment of dysarthric speech is vital for clinical diagnosis and rehabilitation, yet existing automated methods rely heavily on isolated, single-task models or generic speech quality metrics that fail to disentangle heterogeneous neuromotor deficits. This work presents the first systematic investigation into adapting auditory large language models (ALLMs) for multi-dimensional dysarthric speech assessment under a unified, prompt-conditioned framework. Four representative ALLMs are benchmarked across five clinical perceptual dimensions from the Speech Accessibility Project (SAP): naturalness, monopitch, intelligibility, imprecise consonants, and inappropriate silences. To bridge generic perceptual quality and clinical evaluation, a two-stage training paradigm is explored, leveraging out-of-domain pre-training followed by in-domain clinical adaptation. Experimental results demonstrate that leading ALLMs reliably capture fine-grained pathological variations, exhibiting distinct and complementary strengths across prosodic and articulatory attributes. These findings establish ALLMs as a scalable foundation for clinical speech evaluation and provide practical insights into model selection and multi-task transfer potential.
Figures & tables
Figure 1: Distribution of utterances across severity levels (1–7) in the Speech Accessibility Project (SAP) dataset for the five perceptual dimensions: (a) Naturalness, (b) Monopitch, (c) Intelligibility, (d) Imprecise Consonants, and (e) Inappropriate Silences. Higher scores indicate more severe impairment. Notably, several pathological attributes (particularly Inappropriate Silences and Intelligibility) exhibit significant class imbalance toward lower severity levels.
Dimension
Qwen2-Audio
Qwen2.5-Omni
video-SALMONN 2 plus
SALMONN 2
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
Naturalness
0.655
0.684
0.764
0.828
0.836
0.439
0.702
0.738
0.703
0.584
0.644
0.870
Monopitch
0.545
0.534
0.627
0.584
0.595
0.643
0.572
0.567
0.658
0.386
0.406
0.732
Intelligibility
0.549
0.548
0.366
0.610
0.596
0.350
0.635
0.691
0.322
0.592
0.661
0.338
Imprecise Consonants
0.632
0.691
0.449
0.700
0.673
0.411
0.738
0.789
0.342
0.640
0.679
0.456
Inappropriate Silences
0.335
0.371
0.193
0.631
0.750
0.142
0.555
0.616
0.099
0.226
0.298
0.136
Table 1: Comparison of Auditory Large Language Models (ALLMs) on speech quality prediction over the SAP test set. Spearman ( ρ ) and Pearson ( r ) correlations and Mean Absolute Error (MAE) are computed between model predictions and human ratings on five dimensions: naturalness, monopitch, intelligibility, imprecise consonants, and inappropriate silences. Higher ρ and r and lower MAE indicate better performance. Best result per dimension and metric is in bold.
Dimension
Zero-shot
Post-QualiSpeech
Post-SAP
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
Naturalness
0.307
0.327
2.292
0.444
0.437
1.761
0.828
0.836
0.439
Monopitch
0.264
0.267
1.490
0.189
0.190
1.857
0.584
0.595
0.643
Intelligibility
0.328
0.363
1.934
0.150
0.100
2.789
0.610
0.596
0.350
Imprecise Consonants
0.492
0.547
1.714
0.419
0.463
2.215
0.700
0.673
0.411
Inappropriate Silences
− 0.073
− 0.034
3.303
0.098
0.093
3.275
0.631
0.750
0.142
Table 2: Effect of post-training on Qwen2.5-Omni for speech quality prediction over the SAP test set: zero-shot performance (Zero-shot), fine-tuned with QualiSpeech (Post-QualiSpeech) and post-training with SAP (Post-SAP).
Figure 2: Scatter plots of model predictions against human ratings on the SAP test set for (a) Naturalness, (b) Monopitch, (c) Intelligibility, (d) Imprecise Consonants, (e) Inappropriate Silences. These plots visualize the utterance-level predictions corresponding to the Qwen2.5-Omni results reported in Table 1 . Each point represents one test utterance, and the dashed diagonal line indicates perfect agreement between predicted and ground-truth scores.
Dimension
video-SALMONN 2 Plus
Qwen2.5-Omni
Single
Mix
Single
Mix
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
ρ↑
r↑
MAE ↓
Naturalness
0.722
0.758
0.710
0.702
0.738
0.703
0.848
0.858
0.462
0.828
0.836
0.439
Monopitch
0.510
0.507
0.633
0.572
0.567
0.658
0.693
0.687
0.566
0.584
0.595
0.643
Intelligibility
0.592
0.638
0.346
0.635
0.691
0.322
0.669
0.697
0.311
0.610
0.596
0.350
Imprecise Consonants
0.702
0.758
0.414
0.738
0.789
0.342
0.738
0.736
0.385
0.700
0.673
0.411
Table 3: Systematic comparison of dedicated single-task training (Single) versus unified multi-task training (Mix) across representative Auditory Large Language Models over the SAP test set. video-SALMONN 2 Plus demonstrates substantial multi-task synergy where joint training improves performance across almost all dimensions. Qwen2.5-Omni shows that unified multi-task training achieves competitive accuracy (average ρ within 0.046) with only 1/5 of the parameters, while outperforming single-task models on sparse attributes (Inappropriate Silences) and calibration error (Naturalness MAE). Bold indicates the superior result between Single and Mix for each model.
Institute of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland · Department of Mathematics, ETH Zurich, Switzerland · Department of Computational Linguistics, University of Zurich, Switzerland