Inspired by recent developments in neural speech coding and diffusion-based language modeling, we tackle speech enhancement by modeling the conditional distribution of clean speech codes given noisy speech codes using absorbing discrete diffusion. The proposed approach, which we call ADDSE, leverages both the expressive latent space of neural audio codecs and the non-autoregressive sampling procedure of diffusion models. To efficiently model the hierarchical structure of residual vector quantization codes, we propose RQDiT, which combines techniques from RQ-Transformer and diffusion Transformers for non-autoregressive modeling. Results show competitive performance in terms of non-intrusive objective metrics on two datasets, especially at low signal-to-noise ratios and with few sampling steps. Code and audio examples are available online.
Figures & tables
Figure 1: Proposed ADDSE framework.
Figure 2: RQDiT architecture. Two DiTs are applied along the NAC frame and depth dimensions, respectively.
Libri-TUT
Clarity-FSD50K
Params. (M)
PESQ ↑
ESTOI ↑
SDR ↑
MCD ↓
DNSMOS ↑
NISQA ↑
UTMOS ↑
SCOREQ ↑
LPS ↑
SBS ↑
PESQ ↑
ESTOI ↑
SDR ↑
MCD ↓
DNSMOS ↑
NISQA ↑
UTMOS ↑
SCOREQ ↑
LPS ↑
SBS ↑
Noisy
–
1.31
0.61
4.96
6.76
2.92
2.03
2.44
2.47
0.82
0.73
1.34
0.74
4.96
7.52
2.90
2.46
2.42
2.46
0.82
0.69
Clean
–
4.64
1.00
∞
0.00
3.73
3.31
3.06
3.98
1.00
1.00
4.64
1.00
∞
0.00
3.79
3.69
3.38
4.53
1.00
1.00
Clean-NAC
45
2.93
0.89
3.51
2.88
3.75
3.33
3.01
3.90
0.97
0.96
2.85
0.91
5.50
3.08
3.80
3.66
3.31
4.47
0.97
0.96
Conv-TasNet
5
2.44
0.85
15.88
5.03
3.58
3.27
2.68
3.49
0.93
0.85
2.54
0.90
18.18
5.00
3.47
3.32
2.92
3.69
0.91
0.84
BSRNN
13
2.77
0.88
16.77
5.23
3.79
3.59
3.16
3.94
0.96
0.89
3.11
0.94
20.97
4.39
3.72
3.74
3.32
4.22
0.95
0.88
Table 1: Average objective metrics for each model and each dataset.
Figure 3: Top and middle rows: Non-intrusive metrics as a function of Nsteps . Bottom left: Number of function evaluations as a function of Nsteps . Bottom right: DCE on test data.
Figure 4: Non-intrusive metrics as a function of input SNR.
Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.
Diffusion models show potential for speech enhancement but lack linguistic guidance. We condition a diffusion-based model on wav2vec 2.0 features from noisy input, injected at the U-Net bottleneck via Feature-wise Linear Modulation (FiLM). Phonetic representations from wav2vec 2.0 features of degraded speech, anchor the reverse diffusion process. While a frozen wav2vec 2.0 encoder extracts features, a learned FiLM generator produces scale and shift parameters modulating the bottleneck with minimal overhead. Motivated by the optimal Bayesian causal estimator under a linear-Gaussian state-space model, FiLM coefficients are aggregated via exponential smoothing for temporal compression. Evaluation on VoiceBank-DEMAND and LibriMix shows competitive performance against the unconditioned baseline in PESQ, STOI, SI-SDR and DNSMOS. We consistently record an improvement of 0.4 on PESQ score, suggesting self-supervised representations effectively condition diffusion-based speech enhancement.
Shuubham Ojha, Carol Espy-Wilson
Institute for Systems Research, University of Maryland, College Park
We propose Speech Enhancement based on Drifting Models (DriftSE), a novel generative framework that formulates denoising as an equilibrium problem. Rather than relying on iterative sampling, DriftSE natively achieves one-step inference by evolving the pushforward distribution of a mapping function to directly match the clean speech distribution. This evolution is driven by a Drifting Field, a learned correction vector that guides samples toward the high-density regions of the clean distribution, which naturally facilitates training on unpaired data by matching distributions rather than paired samples. We investigate the framework under two formulations: a direct mapping from the noisy observation, and a stochastic conditional generative model from a Gaussian prior. Experiments on the VoiceBank-DEMAND benchmark demonstrate that DriftSE achieves high-fidelity enhancement in a single step, outperforming multi-step diffusion baselines and establishing a new paradigm for speech enhancement.
Liang Xu, Diego Caviedes-Nozal, W. Bastiaan Kleijn +2
Victoria University of Wellington, 3 Lincoln University, New Zealand · GN Advanced Science, Denmark
Real-world speech is often corrupted by multiple degradations simultaneously, including additive noise, reverberation, and nonlinear distortion. Diffusion-based enhancement methods perform well on single degradations but struggle with compound corruptions. Prior noise-aware approaches inject conditioning at the input layer only, which can degrade performance below that of an unconditioned model. To address this, we propose injecting degradation conditioning, derived from a pretrained encoder with multi-task heads for noise type, reverberation, and distortion, into the timestep embedding so that it propagates through all residual blocks without architectural changes. In controlled experiments where only the injection method varies, input-level conditioning performs worse than no encoder at all on compound degradations, while layer-wise injection achieves the best results. The method also generalizes to diverse real-world recordings.