Real-world speech is often corrupted by multiple degradations simultaneously, including additive noise, reverberation, and nonlinear distortion. Diffusion-based enhancement methods perform well on single degradations but struggle with compound corruptions. Prior noise-aware approaches inject conditioning at the input layer only, which can degrade performance below that of an unconditioned model. To address this, we propose injecting degradation conditioning, derived from a pretrained encoder with multi-task heads for noise type, reverberation, and distortion, into the timestep embedding so that it propagates through all residual blocks without architectural changes. In controlled experiments where only the injection method varies, input-level conditioning performs worse than no encoder at all on compound degradations, while layer-wise injection achieves the best results. The method also generalizes to diverse real-world recordings.
Figures & tables
Figure 1: Overview of SLICE. (a) The encoder produces h∈Rdh ; branch projections are concatenated and mapped to cextra∈Rd . (b) cextra is added to temb and propagated through every residual block. Dashed orange: auxiliary heads (train-only). dw=768 , dh=256 , db=128 , d=512 .
Noise-Only
Multi-Degradation
Method
PESQ
ESTOI
SDR
UTMOS
PESQ
ESTOI
SDR
UTMOS
MP-SENet [ 19 ]
3.16
0.88
19.6
3.90
2.39
0.62
− 0.3
3.00
MetricGAN+ [ 20 ]
3.13
0.83
0 8.6
3.63
2.53
0.62
− 5.5
2.82
SGMSE+ [ 1 ]
2.94
0.87
17.4
3.83
2.30
0.63
− 0.0
2.99
SGMSE+ (dereverb) [ 1 ]
1.98
0.78
10.9
3.16
1.99
0.66
− 3.3
2.96
NASE [ 12 ]
2.75
0.85
17.3
3.83
2.22
0.64
− 0.5
2.99
Table 1: Speech enhancement results on noise-only (824 files) and multi-degradation (2472 files) conditions. All baselines are trained on noise-only data. Controlled ablations are in Table 2 . Best per condition in bold . SDR denotes SI-SDR (dB).
Variant
PESQ
ESTOI
SDR
UTMOS
SLICE
2.60
0.80
3.7
3.71
w/o aux. loss ( λ=0 )
2.58
0.77
2.2
3.63
Input addition
2.49
0.73
1.4
3.62
No encoder
2.60
0.77
2.3
3.70
Zero conditioning
2.14
0.67
− 1.6
3.04
Uniform weights
2.60
0.80
3.7
3.71
Table 2: Results of the ablation study evaluated on the multi-degradation test set. The term “addition” refers to the input-level addition method introduced by [ 12 ] .
Type
N
PESQ
ESTOI
SDR
UTMOS
Noise only
824
2.82
0.86
17.5
3.93
Noise+Reverb
330
1.72
0.65
− 17.2
3.31
Noise+Distort
318
2.84
0.86
15.8
3.94
Noise+Reverb+Distort
338
1.74
0.65
− 18.4
3.34
Reverb only
341
2.01
0.74
− 17.1
3.42
Distort only
321
4.21
0.98
23.3
4.05
Table 3: Detailed performance breakdown across six distinct degradation categories within the multi-degradation test set.
Dataset
Model
PESQ
ESTOI
UTMOS
VOiCES
SGMSE+ (pretrained)
1.20
0.34
1.98
SGMSE+ (no enc.)
1.59
0.65
2.83
SLICE
1.56
0.67
2.77
DAPS
SGMSE+ (pretrained)
2.10
0.69
2.96
SGMSE+ (no enc.)
2.04
0.65
2.96
SLICE
2.20
0.60
3.32
Table 4: Evaluation on in-the-wild dataset. SGMSE+ (pretrained) is the publicly available denoise model [ 1 ] ; SGMSE+ (no enc.) is trained from scratch on the same multi-degradation data as SLICE but without an encoder. SI-SDR is omitted as it is unreliable for reverberant recordings.
Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.
Using speaker embeddings as conditioning can strengthen speech enhancement, but most methods either require clean enrollment audio or rely on embeddings extracted from noisy speech, which are fragile under noise and domain shift. We propose G-MaP-SE, a guided enhancement framework that builds a clean-speech embedding prior with a Gaussian Mixture Model (GMM) and refines a noisy conditioning embedding by matching it to this prior. The matched prior embedding is then injected into a time-frequency enhancement backbone via a lightweight gated fusion module. Experiments on VoiceBank+DEMAND and DNS Challenge 2020 datasets show that the proposed prior matching consistently outperforms noisy conditioning and substantially narrows the gap to an oracle clean-conditioning upper bound, while requiring no enrollment audio at inference time. The code, audio samples, and checkpoint are available.
Inspired by recent developments in neural speech coding and diffusion-based language modeling, we tackle speech enhancement by modeling the conditional distribution of clean speech codes given noisy speech codes using absorbing discrete diffusion. The proposed approach, which we call ADDSE, leverages both the expressive latent space of neural audio codecs and the non-autoregressive sampling procedure of diffusion models. To efficiently model the hierarchical structure of residual vector quantization codes, we propose RQDiT, which combines techniques from RQ-Transformer and diffusion Transformers for non-autoregressive modeling. Results show competitive performance in terms of non-intrusive objective metrics on two datasets, especially at low signal-to-noise ratios and with few sampling steps. Code and audio examples are available online.
Philippe Gonzalez
Department of Health Technology, Technical University of Denmark
Diffusion models show potential for speech enhancement but lack linguistic guidance. We condition a diffusion-based model on wav2vec 2.0 features from noisy input, injected at the U-Net bottleneck via Feature-wise Linear Modulation (FiLM). Phonetic representations from wav2vec 2.0 features of degraded speech, anchor the reverse diffusion process. While a frozen wav2vec 2.0 encoder extracts features, a learned FiLM generator produces scale and shift parameters modulating the bottleneck with minimal overhead. Motivated by the optimal Bayesian causal estimator under a linear-Gaussian state-space model, FiLM coefficients are aggregated via exponential smoothing for temporal compression. Evaluation on VoiceBank-DEMAND and LibriMix shows competitive performance against the unconditioned baseline in PESQ, STOI, SI-SDR and DNSMOS. We consistently record an improvement of 0.4 on PESQ score, suggesting self-supervised representations effectively condition diffusion-based speech enhancement.
Shuubham Ojha, Carol Espy-Wilson
Institute for Systems Research, University of Maryland, College Park