Hands-free communication devices and speakerphones are inherently affected by acoustic echo and background noise. To mitigate these impairments, end-to-end discriminatively trained neural networks have emerged as the best-performing approach in research and deployment. While recent advancements in generative methods have provided remarkable results for various speech enhancement tasks, diffusion-based acoustic echo control (AEC) research is still restricted to non-causal, utterance-level processing, thereby not widely applicable in practice. With this work, we are the first to propose low-delay (i.e., causal) diffusion-based joint AEC and noise control models DiffVQE2 / DiffVQE2-S, excelling the so-far state of the art DeepVQE / DeepVQE-S models in multiple objective metrics, and, most importantly, in subjective MOS, respectively. In addition, our models are less complex. Furthermore, we show that a limited lookahead applied to the efficient DiffVQE2-S model allows for an even higher performance. These results have been obtained on the ICASSP 2023 AEC Challenge blind test set. We claim the first streaming-capable, diffusion-based acoustic echo and noise control that excels state-of-the-art discriminative approaches.
Figures & tables
Fig. 3 : Influence of lookahead on Dval performance in all conditions.
Method
#Param. (M)
#GFLOPS
Alg.
Look-
DT
DT
DT
DT
DT
STFE
STNE
STNE
STNE
STNE
Avg.
DT
DT
STFE
STNE
Avg.
Delay
ahead
Echo
Other
OVRL
SIG
BAK
Echo
Other
OVRL
SIG
BAK
Rank ↓
MOS E
MOS O
MOS E
MOS O
MOS
Unprocessed
-
-
-
-
1.76
4.10
2.12
3.10
2.23
2.07
3.79
2.75
3.33
3.37
-
2.10
2.87
2.52
3.45
2.74
DiffVQE [ 19 ]
5.1
5.37
∞
∞
4.62
4.11
2.93
3.25
3.96
4.43
4.26
3.14
3.43
4.07
-
3.70
3.54
3.71
3.77
3.68
DeepVQE [ 8 ]
5.3
42.24
32ms
0
4.64
3.84
2.75
3.11
3.86
4.37
3.93
2.97
3.31
4.03
1.2
3.66
3.26
3.48
3.34
3.44
DiffVQE2
5.1
5.37
32ms
0
4.48
3.76
2.67
3.00
3.86
4.46
3.88
2.98
3.28
4.02
1.7
3.76
3.39
3.77
3.39
3.58
Table 1: Model footprint, algorithmic delay, performance on the ICASSP 2023 AEC Challenge test set Dtest . Separate rankings (large/small).
Inspired by recent developments in neural speech coding and diffusion-based language modeling, we tackle speech enhancement by modeling the conditional distribution of clean speech codes given noisy speech codes using absorbing discrete diffusion. The proposed approach, which we call ADDSE, leverages both the expressive latent space of neural audio codecs and the non-autoregressive sampling procedure of diffusion models. To efficiently model the hierarchical structure of residual vector quantization codes, we propose RQDiT, which combines techniques from RQ-Transformer and diffusion Transformers for non-autoregressive modeling. Results show competitive performance in terms of non-intrusive objective metrics on two datasets, especially at low signal-to-noise ratios and with few sampling steps. Code and audio examples are available online.
Philippe Gonzalez
Department of Health Technology, Technical University of Denmark
Text-to-audio (T2A) diffusion models generate high-quality audio but require tens of neural function evaluations (NFEs), resulting in substantial inference latency. Existing acceleration methods primarily optimize each generation trajectory independently, even when reusable acoustic structure exists in prior examples. We introduce SoundWeaver, the first training-free framework for compositional cross example acoustic reuse. Rather than generating from pure noise, SoundWeaver constructs a prompt-conditioned acoustic prior from segments distributed across cached audios and warm-starts diffusion from an intermediate noise level. We introduce an Acoustic Composition Graph (ACG), which uses neural codec representations and segment-level identification to identify candidate cross-cache transitions and structured path search to jointly optimize prompt relevance and acoustic continuity. We further establish a warm-start error bound showing how prior quality and diffusion SNR jointly govern the admissible skip depth, motivating an adaptive contextual-bandit Skip Gater that selects the warm-start level for each request. Across T2A diffusion backbones, SoundWeaver achieves 1.48-3.74x generation speedup with only a 1K-clip cache while improving generation quality.
Ayush Barik, Sofia Stoica, Nikhil Sarda +4
University of Illinois Urbana-Champaign · Assured Intelligence
Generative audio requires fine-grained controllable outputs, yet most existing methods require model retraining on specific controls or inference-time controls (e.g., guidance) that can also be computationally demanding. By examining the bottlenecks of existing guidance-based controls, in particular their high cost-per-step due to decoder backpropagation, we introduce a guidance-based approach through selective TFG and Latent-Control Heads (LatCHs), which enables controlling latent audio diffusion models with low computational overhead. LatCHs operate directly in latent space, avoiding the expensive decoder step, and requiring minimal training resources (7M parameters and ≈ 4 hours of training). Experiments with Stable Audio Open demonstrate effective control over intensity, pitch, and beats (and a combination of those) while maintaining generation quality. Our method balances precision and audio fidelity with far lower computational costs than standard end-to-end guidance. Demo examples can be found at https://zacharynovack.github.io/latch/latch.html.