DAMOS: Learning Distortion-Aware Speech Quality Assessment through Explicit Distortion Localization
Authors: Naiyuan Li, Li Dong, Diqun Yan
Organizations: College of Information Science and Engineering, Ningbo University, Ningbo, China · College of Artificial Intelligence, Ningbo University of Finance and Economics, Zhejiang Provincial Collaborative Innovation Center for Digital Supply Chain and Artificial Intelligence of Bulk Commodities
Automatic speech quality assessment aims to predict Mean Opinion Scores (MOS) consistent with human subjective perception and is essential for evaluating speech generation, enhancement, and communication systems. For speech signals, especially synthetic speech, distortions often occur locally, and overall perceptual quality is usually dominated by a small number of perceptually salient distortion regions. However, most existing methods are primarily optimized with utterance-level MOS, which provides only coarse-grained supervision and offer no explicit indication of where perceptually important distortions occur. To address this limitation, we introduce explicit distortion localization as auxiliary knowledge for speech quality assessment. We construct the first partially distorted speech dataset with frame-level distortion annotations and train a localization model to generate distortion cues. Building on these cues, we propose DAMOS, a distortion-aware speech quality assessment framework that integrates localization information into the MOS prediction pipeline. Experiments on multiple public benchmarks demonstrate that DAMOS consistently outperforms existing methods and exhibits strong cross-dataset generalization, validating the effectiveness of explicit distortion localization for speech quality assessment.
Figures & tables
Class
Type
Samples
Ratio (%)
–
Clean
406
2.7
A1
Codec (single)
1,327
8.8
A2
Codec tandem
693
4.6
A3
Low-bitrate compression
871
5.8
A4
Resampling
565
3.8
A5
Reverberation
1,411
9.4
TABLE I: Distortion type distribution in the synthetic dataset (actual generated statistics).
Fig. 1: Architecture of the proposed DAMOS.
Method
PSTN
Tencent
NISQA_FOR
NISQA_LIVETALK
NISQA_P501
SRCC
LCC
SRCC
LCC
SRCC
LCC
SRCC
LCC
SRCC
LCC
NISQA
0.819
0.818
0.943
0.946
0.876
0.891
0.755
0.778
0.907
0.904
SSL-MOS
0.816
0.812
0.959
0.960
0.859
0.857
0.674
0.696
0.902
0.895
MOSA-Net+
0.816
0.812
0.936
0.937
0.863
0.863
0.633
0.633
0.876
0.887
DNSMOS Pro
0.799
0.797
0.918
0.923
0.754
0.760
0.519
0.512
0.864
0.824
SLL-Conformer [ 39 ]
0.825
0.822
0.952
0.954
0.904
0.914
0.759
0.778
0.926
0.928
TABLE III: performance comparison on the NISQA, PSTN and Tencent datasets.
Model
TCD-VoIP [ 8 ]
NISQA_P501
NISQA_LIVETALK
NISQA_FOR
Tencent
PSTN
SOMOS [ 11 ]
Avg.
LDNet
0.734
0.611
-0.092
0.393
0.328
0.578
-0.030
0.360
NISQA
0.561
0.519
0.236
0.359
0.241
0.433
0.111
0.351
SSL-MOS
0.750
0.847
0.680
0.737
0.675
0.736
0.435
0.694
UTMOS
0.775
0.890
0.792
0.798
0.714
0.749
0.307
0.718
MOSA-Net+
0.828
0.816
0.629
0.685
0.755
0.736
0.387
0.691
DNSMOS Pro
0.554
0.584
0.147
0.275
0.465
0.576
0.055
0.379
TABLE IV: Cross-corpus generalization results: SRCC comparison across datasets.
Method
BVCC
TCD-VoIP
NISQA_P501
NISQA_LIVETALK
NISQA_FOR
Tencent
PSTN
SOMOS
Avg
Last Hidden Layer
0.853
0.735
0.826
0.333
0.480
0.687
0.630
0.380
0.616
Best Hidden Layer
0.887
0.851
0.827
0.727
0.767
0.719
0.740
0.405
0.740
Average of All Layers
0.876
0.815
0.855
0.759
0.825
0.791
0.742
0.399
0.758
DAMOS (DSLA)
0.885
0.860
0.856
0.781
0.808
0.742
0.738
0.435
0.763
TABLE VI: Analysis of different hidden-layer aggregation strategies. All models evaluated using utterance-level SRCC.
Fig. 2: Average normalized layer weights learned by the proposed Distortion-Sensitive Layer Adaptation (DSLA) on different speech quality assessment datasets.
Class
Frame-level
Boundary
EER(%)
F1
Acc
Prec
Rec
EER(%)
F1
Acc
Prec
Rec
all
15.77
0.807
0.903
0.963
0.695
13.61
0.742
0.976
0.887
0.638
A1
16.40
0.819
0.906
0.976
0.705
12.74
0.762
0.979
0.909
0.656
A2
14.57
0.775
0.894
0.965
0.648
16.35
0.683
0.972
0.835
0.578
A3
12.87
0.838
0.920
0.944
0.753
12.79
0.802
0.981
0.934
0.703
A4
19.27
0.767
0.897
0.953
0.642
9.47
0.793
0.983
0.873
0.727
TABLE VII: Results of Per-distortion-class frame-level and boundary detection metrics
Dataset
Mask Condition
MSE
SRCC
LCC
BVCC
Normal mask
0.191
0.885
0.884
Zero mask
0.191
0.884
0.884
One mask
0.227
0.870
0.872
TCD-VoIP
Normal mask
0.300
0.837
0.858
Zero mask
0.304
0.829
0.858
One mask
0.374
0.818
0.845
TABLE VIII: Effect of distortion localization guidance under different mask conditions.
Explainable speech quality assessment requires moving beyond Mean Opinion Scores (MOS) to analyze underlying perceptual dimensions. To address this, we introduce a novel post-training method that tailors the foundational Audio Large Language Model for multidimensional reasoning, detection and classification of audio artifacts. First, a calibration stage aligns the model to predict predefined perceptual dimensions. Second, a reinforcement learning stage leverages Group Relative Policy Optimization (GRPO) with dimension-specific rewards to heavily enhance accuracy of descriptions and temporal localization of quality issues. With this approach we reach state-of-the-art results of 0.71 mean PCC score on the multidimensional QualiSpeech benchmark and 13% improvement in MOS prediction driven by RL-based reasoning. Furthermore, our fine-grained GRPO rewards substantially advance the model's ability to pinpoint and classify audio artifacts in time.
UTMOS has become one of the most commonly used deep neural network-based speech quality assessment (SQA) metrics in speech processing research. In this paper, we attack UTMOS to probe its robustness. Starting from high-quality speech samples, we optimize the input in two directions: a score-preserving attack, which degrades perceived quality while maintaining the predicted score, and a quality-preserving attack, which lowers the predicted score while maintaining perceived quality. We consider three input spaces: raw waveform, mel spectrogram with a HiFi-GAN vocoder, and the latent space of EnCodec, a neural audio codec. Experimental results show that score-preserving attacks are effective against UTMOS. Although perfect quality-preserving attacks are more difficult, optimization in the EnCodec latent space provides the best chance of success. These results reveal failure modes of UTMOS and highlight the importance of robustness analysis for DNN-based SQA metrics.
Localising artifacts in synthetic speech remains challenging, as most evaluation methods yield only global quality scores. This paper presents XSQ-AST, a framework that combines the SQ-AST speech quality model with WhisperX phoneme alignment and multiple saliency methods to produce temporally localised artifact diagnostics without model retraining. Saliency maps are projected onto continuous distributions via kernel density estimation and onto phoneme boundaries via phoneme-discretised saliency maps. A 40-participant listening test validated the framework across five perceptual dimensions. Attention Rollout, Attention Flow and an adapted GradCAM produced temporal distributions that correlated with listener highlights, with different methods best suited to different artifact types. An AUC-ROC analysis confirmed discrimination above chance.
Ben Heritage, Luca Resti, Mónica Villanueva Aylagas +3
AudioLab, School of Physics, Engineering and Technology, University of York, United Kingdom · Department of Computer Science, University of York, United Kingdom · SEED – Electronic Arts (EA), Sweden +1