Speech Enhancement

Ask about this topic

Overview

Speech enhancement seeks to recover clear, intelligible speech from recordings degraded by background noise, reverberation, echo, or other speakers. Methods range from spectral masking and multichannel beamforming, such as MVDR, to neural time-domain models and newer diffusion, flow-matching, and codec-based generators; visual cues and speaker references can help identify the target. Systems are commonly tested on benchmarks such as VoiceBank+DEMAND, DNS, and CHiME, while real-world meetings, hearing aids, and mobile devices bring challenges in domain shift, latency, and power use. A central tension is that better perceptual scores do not always mean more accurate speech recognition, and generative methods must improve quality without changing the speaker’s words or introducing artifacts.

Latest papers

CardsList

128 papers15 papers in the last four weeks, down 50% on the four weeks before. 0.1% of all new papers.

Aug 1, 2026

AnyBand: Unified Multi-Bandwidth Speech Extension via Frequency-Aware In-Context Spectral Infilling

Bandwidth extension (BWE) aims to recover missing high-frequency content from band-limited speech. Existing methods often formulate BWE as a fixed or predefined bandwidth conversion problem, potentially requiring cutoff-specific models or retraining when the input bandwidth changes. This assumption limits their applicability to practical scenarios where speech may arrive with diverse cutoff frequencies. We propose AnyBand, a unified BWE framework that recasts bandwidth extension as in-context spectral infilling. Motivated by prompt-based zero-shot speech generation, AnyBand conditions high-frequency generation on the observed low-frequency spectrum, using the available band as a frequency-domain prompt that conveys content, speaker, prosodic, and spectral-envelope cues. This formulation enables a single model to perform cutoff-conditioned generation over a continuous range of input bandwidths. AnyBand is trained with missing-band conditional flow matching and an Easy-to-Balanced cutoff curriculum over continuously sampled cutoff frequencies. To better exploit the spectral prompt, we introduce a frequency-aware Diffusion Transformer that models cross-frequency interactions and long-range temporal dependencies, followed by a physically motivated multi-view adversarial refinement stage to enhance spectral realism, envelope coherence, and harmonic consistency. Experiments on multiple datasets and bandwidth settings show that AnyBand consistently improves spectral reconstruction over existing baselines while achieving competitive perceptual quality across both standard and irregular input cutoffs. Audio samples are available.

Junchuan Zhao, Minh Duc Vu, Bowen Zhang +1

School of Computing, National University of Singapore · Department of Statistics & Data Science, National University of Singapore · College of Computing and Data Science, Nanyang Technological University

Audio Diffusion ModelsSpeech Enhancement
AskPDF
Jul 28, 2026

faster-enhancer.c: A Dependency-Free int8 Runtime for Streaming Speech Enhancement on Commodity CPUs

This is an implementation and measurement study of what it costs to run a streaming speech enhancer on a CPU. We port FastEnhancer-Medium at 48 kHz to faster-enhancer.c, a C runtime with six int8 GEMM tiers selected at initialization, leaving architecture and weights untouched. One Apple M2 core reaches 0.069 real-time factor, against 0.230 for the fp32 ONNX Runtime graph on the same machine, a 3.3x speedup. A Galaxy S23+ (Snapdragon 8 Gen 2) reaches 0.096. The speedup comes from specializing every layer of the runtime around one fixed model. Activation ranges are recomputed per frame, so no calibration set is needed; the k=3 convolutions use Winograd F(2,3); cross-stage state is fp16; the GRU and the dequantization epilogues are fused; and nothing is allocated after startup. Over 824 VoiceBank-DEMAND utterances the engine tracks fp32 to within -0.006 PESQ and -0.08 dB SNR. Speed alone does not settle deployment cost. The enhancer holds a fraction of a core for as long as the microphone is open, so its real-time factor is a duty cycle. A benchmark races through a file; an audio callback does not. Pacing to the 6.67 ms deadline costs 4.2x per frame, saves 49% of the energy, and leaves the cheapest core placement missing 96% of its deadlines. All SIMD tiers within an architecture family emit byte-identical output. The runtime is released as a dependency-free library.

Gyeongmin Kim

Department of Computer Science, Hanyang University, Seoul, Korea

Efficient Neural Network InferenceSpeech EnhancementOn-Device Inference
AskPDF
Jul 21, 2026

Towards Array-Invariant Speech Enhancement via Geometry-Aware Dynamic Convolution

Multi-channel speech enhancement (SE) systems exhibit superior performance over single-channel methods but are constrained to fixed microphone array configurations. This restricts their real-world deployment across devices with diverse array geometries. While recent array-agnostic SE methods address variable microphone numbers and permutations, they largely fail to exploit explicit array geometry priors when available, missing a crucial cue for optimal spatial filtering. A Geometry-Aware Dynamic Convolution (Geo-DConv) framework is proposed, which explicitly leverages microphone coordinates to transform standard fixed-array SE models into robust array-invariant systems. Experiments are conducted on the recent real-recorded RealMAN multi-channel speech dataset. Results demonstrate that the proposed architecture enables two widely used fixed-array models to adapt to array-invariant settings, with consistent performance improvements across diverse array topologies.

Zhenglong Liu, Wangyou Zhang, Chenda Li +1

Auditory Cognition and Computational Acoustics Lab Shanghai Jiao Tong University, Shanghai, China · VUI Labs

Dynamic ConvolutionSpeech Enhancement
AskPDF
Jul 21, 2026

CAPS: A Cascaded Reconstruction Model to Power Saving in Hearables Using Sub-Nyquist Sampling with Bandwidth Extension

Hearables are wearable computers worn on the ear. Bone conduction microphones are used with air conduction microphones in hearables for multimodal speech enhancement in noisy conditions. Despite this potential, current models largely fail to explore how jointly reducing sampling bit resolution and sampling frequency in analog-to-digital converters (ADCs) of hearables impacts both power usage and audio quality. Furthermore, current frameworks cannot do sub-Nyquist sampling in hearables because they lack a method to reconstruct wideband signals from narrowband components. We therefore propose CAPS, which (i) intentionally employs sub-Nyquist sampling and low bit resolution in ADCs, achieving a 3.3x reduction in power consumption in hearables, and (ii) supports streaming operation on mobile platforms with an inference time of 1.36 ms and a memory footprint of 11.04 MB. CAPS ensures robust speech intelligibility in real-world settings, bridging the gap between efficiency and power savings.

Tarikul Islam Tamiti, Sajid Fardin Dipto, Luke Baja-Ricketts +2

Cyber-Security Engineering, George Mason University, USA

Energy-Efficient MLSpeech Enhancement
AskPDF
Jul 9, 2026

It Takes Few to TANGO: A Quantized Distributed Model for Binaural Speech Enhancement

Neural network-based multichannel speech enhancement systems achieve strong enhancement performance, but their computational and memory requirements limit deployment on resource-constrained devices. This paper investigates low-precision inference for TANGO, a hybrid distributed binaural speech enhancement system combining neural mask estimation with spatial filtering. We evaluate post-training quantization and quantization-aware training for the neural components, and analyze how quantization errors in the mask estimators propagate through the downstream spatial filtering stage. Our analysis shows that, although quantization degrades intermediate mask estimates, the spatial filtering stage compensates for most quantization-induced errors. Leveraging this robustness, we simplify TANGO into MN-TANGO, reducing both model size and computational complexity while maintaining comparable final performance. By combining INT8 weight-and-activation quantization with ERB compression and grouped recurrent layers, the most compact MN-TANGO reaches 4.65 MMAC/s and 0.177 MB.

Zahra Benslimane, Pierre Chouteau, Martyna Poreba +4

Université Paris-Saclay, CEA, List, F-91120 Palaiseau, France · Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France

Quantization-Aware TrainingSpeech EnhancementPost-Training QuantizationLow-Bit QuantizationNeural Network QuantizationNeural Network Compression
AskPDF
Jul 6, 2026

Ranking the Impact of Contextual Specialization in Neural Speech Enhancement

We systematically investigate neural speech enhancement systems, ranging from very small (∼\sim10,k parameters) to medium-large (∼\sim2-5,M parameters), which specialize to acoustic conditions using contextual information such as speaker identity, noise type, speaker gender, spoken language, and SNR. By fine-tuning generalist models on specific data subsets, we find that specializing to a speaker's identity consistently yields the largest gains in estimated speech intelligibility and quality. In contrast, specializing to SNR, noise type, or gender offers only marginal benefits. Crucially, we show that a small model specialized to both a specific speaker and a specific noise type can match or exceed the performance of a generalist model ten times its size. Further, cross-lingual tests reveal that models specialized to a target language outperform multilingual generalists, suggesting that language is a salient feature for specialization. These findings highlight the potential of small, adaptive models for resource-constrained applications like hearing aids, which specialize on-the-fly to contextual information.

Peter Leer, Svend Feldt, Zheng-Hua Tan +2

Eriksholm Research Centre, Snekkersten, Denmark · Aalborg University, Department of Electronic Systems, Aalborg, Denmark

Speech Enhancement
AskPDF
Jul 5, 2026

Weakly Guided and Autoregressive Beamformer Parameterization for Generalizable Moving Speaker Extraction in Higher-Order Ambisonics

Linear spatial filters (beamformers) enable robust, generalizable and interpretable speech enhancement with performance guarantees under ideal parameterization. Modern beamformers are often parameterized by deep neural networks, whose performance degrades in dynamic scenarios with multiple moving speakers of unknown directions. We propose a data-driven beamforming pipeline, which only requires an estimate of the target's initial direction. Building on a higher-order ambisonics representation, we show that neural temporal-spectral processing can be decoupled from linear spatial processing, and thereby achieve generalizable and array-agnostic enhancement. By incorporating autoregression into a frame-wise causal framework, we maintain consistent performance throughout fast speaker motion and long recordings. Evaluation on synthetic data demonstrates robust enhancement under challenging conditions with closely spaced and crossing speakers. Real-world recordings in a dynamic office meeting scenario complement these findings and show generalizability across varying ambisonics orders.

Jakob Kienegger, Tal Peer, Sina Khanagha +1

Signal Processing (SP) Group, University of Hamburg

Speech EnhancementSpatial AudioMicrophone Array Processing
AskPDF
Jul 2, 2026

RT-Tango: Real-Time Distributed Binaural Speech Enhancement for Low-Power Hearing Aid Devices

Real-time binaural speech enhancement is constrained by latency, computational cost, and inter-device communication, yet existing efficient solutions predominantly address single-channel settings. In this paper, we introduce RT-Tango, a real-time distributed binaural speech enhancement framework designed for streaming on resource-constrained platforms and specifically for hearing aids. RT-Tango relies on a two-stage distributed architecture combining perceptually motivated ERB feature compression, lightweight grouped recurrent mask estimation, and temporal sparsification to reduce computational cost. Stringent latency constraints are addressed by decoupling spectral resolution from algorithmic delay using an asymmetric STFT, together with causal recurrent inference and online estimation of spatial statistics. Experimental results show that RT-Tango achieves competitive speech enhancement while significantly reducing MACs operations and functioning at ultra-low latencies as low as 8 ms.

Z. Benslimane, P. Chouteau, M. Poreba +4

Universit´e Paris-Saclay, CEA, List, F-91120 Palaiseau, France · Universit´e de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France

Speech ProcessingSpeech EnhancementStreaming Algorithms
AskPDF
Jun 30, 2026

Improving multichannel speech enhancement through accurate room-acoustic simulations

Room-acoustic simulations are widely used to augment training data for deep-learning-based speech enhancement. While most pipelines rely on simplified geometrical acoustics, wave-based approaches offer greater physical accuracy. In this work, we examine how simulation fidelity affects multichannel speech enhancement performance. To this end, we train SpatialNet on datasets augmented with different room-acoustic simulation methods and evaluate the resulting models on measured data. We compare lower-fidelity datasets based on geometrical acoustics with a high-fidelity dataset using advanced acoustic modelling and a hybrid combination of wave-based and geometrical acoustics simulations. Training on the high-fidelity dataset results in an up to 38 % relative reduction in median word error rate compared to the lower-fidelity alternatives. These results show that augmentation with high-fidelity room-acoustic simulations directly translates into improved multichannel speech enhancement performance.

Georg Götz, Alessia Milo, Steinar Guðjónsson +3

Treble Technologies, Reykjavík, Iceland

Synthetic Data AugmentationSpeech EnhancementPhysics-Based SimulationRoom Acoustics
AskPDF
Jun 24, 2026

SE-AGCNet: An End-to-End Framework for Joint Speech Enhancement and Loudness Control in Meeting Scenarios

Conventional audio pipelines typically treat speech enhancement (SE) and automatic gain control (AGC) as discrete modules, which often limits overall performance. For instance, applying AGC before SE may inadvertently amplify background noise, while prioritizing SE tends to over-suppress low-volume speech. To address these limitations, we propose SE-AGCNet, an end-to-end framework that jointly optimizes SE and AGC. Tailored for meeting scenarios with significant volume variations, SE-AGCNet leverages the synergy between the two tasks: SE preserves quiet speech, thereby facilitating effective volume adjustment by the AGC component. Furthermore, we propose a specialized data simulation pipeline, SE-AGC-DataGen, and incorporate standardized loudness evaluation metrics: integrated loudness (LUFS), short-term loudness (St LUFS), and LRA. Experiments show that SE-AGCNet consistently achieves target loudness while improving speech quality and ASR accuracy over competitive baselines.

Jinming Zhang, Wei Rao, Xionghu Zhong +1

Zhejiang University, China · Nanyang Technological University, Singapore · Hunan University, China

Speech ProcessingSpeech Enhancement
AskPDF
Jun 24, 2026

One Model, Many Latencies: Universal Speech Enhancement for Diverse Real-Time Applications

Different real-time speech applications impose distinct latency budgets, often requiring separately trained enhancement models for each scenario. In this paper, we propose a one-for-all, real-time universal speech enhancement model that provides explicit control over both algorithmic and computational latency. Algorithmic latency is flexibly adjusted via configurable look-ahead frames. To avoid learning inefficiency caused by varying padding configurations, we introduce parallel convolutional layers corresponding to different look-ahead settings. Computational latency is controlled through an early-exit mechanism, enabling inference at different network depths. To narrow the performance gap between specialized and flexible models, we propose a two-stage training strategy with a shared-to-multiple decoder transition. Overall, the proposed framework enables a single model to be deployed across diverse latency budgets without retraining separate models. Model weights are available for download at: https://huggingface.co/nvidia/Real-time_RE-USE

Szu-Wei Fu, Rong Chao, Xuesong Yang +4

NVIDIA · Academia Sinica, Taipei, Taiwan

Speech Enhancement
AskPDF
Jun 23, 2026

Beyond U-Net: A Latent-Representation-Aligned Skip-Free Backbone for Flow-Matching Speech Enhancement

Generative models, particularly diffusion and score-based approaches, have recently achieved strong performance in speech enhancement, but their iterative sampling process limits real-time deployment. Flow Matching offers an efficient alternative by transporting noisy speech toward clean speech through an ordinary differential equation with few function evaluations. In this work, we propose a skip-free encoder-decoder backbone for flow-matching speech enhancement, guided by Latent Representation Alignment (LRA). Instead of relying on U-Net skip connections, which may transfer noise-correlated low-level features to the decoder, the proposed model aligns its bottleneck and decoder representations with clean latent features extracted from a frozen Descript Audio Codec encoder-decoder without quantization. This codec-aligned supervision promotes compact clean-speech representations while preserving efficient few-step inference. Experiments on WSJ0-CHiME3 and VoiceBank-DEMAND show improved PESQ and perceptual quality, especially on VoiceBank-DEMAND, using only five function evaluations.

Wangyi Pu, Michele Scarpiniti

Department of Information Engineering, Electronics and Telecommunications, Sapienza University of Rome, Italy

Flow MatchingSpeech EnhancementFlow-Based Generative ModelingLatent Space Alignment
AskPDF
Jun 23, 2026

Joint Learning of Covariance Estimation and White Noise Gain for Robust MVDR Beamforming

The minimum variance distortionless response (MVDR) beamformer is widely used for multichannel speech enhancement due to strong noise suppression while preserving target signals. In practice, its performance is sensitive to microphone self-noise and array mismatches. Existing approaches typically rely on fixed, manually tuned WNG thresholds or diagonal loading, leading to suboptimal performance under unknown or time-varying acoustic conditions. This paper proposes a data-driven MVDR framework that adaptively estimates the WNG constraint using a deep neural network. The network jointly predicts a time-frequency noise mask for covariance estimation and a frequency-dependent WNG threshold, enabling dynamic robustness-directivity control. A differentiable robust MVDR layer is integrated into the framework, allowing end-to-end optimization. Experiments demonstrate consistent improvements in speech quality and intelligibility over conventional fixed-WNG MVDR methods.

Yongyi Deng, Hanchen Pei, Jianbo Ma +3

School of Electronic Information, Wuhan University, Wuhan, Hubei, China · Dolby Laboratories · CIAIC, Northwestern Polytechnical University, Xi’an, Shaanxi, China +1

Speech EnhancementNoise Robustness
AskPDF
Jun 22, 2026

Neuromorphic Speech Enhancement with Dual-Branch Spiking Neural Networks

Spiking neural network (SNN)-based neuromorphic speech enhancement has emerged as a promising paradigm due to its energy efficiency, yet it still underperforms classical artificial neural network (ANN)-based approaches owing to binary activations and the lack of well-designed network architectures. To overcome this limitation, we propose a novel dual-branch spiking neural network architecture equipped with a gated spiking unit (GSU), termed GSU-DBNet. Specifically, GSU-DBNet simultaneously models the speech magnitude spectrum and complex spectrum, predicting the corresponding magnitude and complex spectral masks. Meanwhile, a dual-path GSU module is adopted to exploit temporal and frequency information for enhanced spatiotemporal feature representation. Experiments on a popular benchmark dataset show that GSU-DBNet achieves a PESQ score of 3.04 with only 394K parameters, outperforming existing SNN-based methods while using only 4.5%--10.6% of the parameters of representative ANN-based models.

Taiyu Meng, Wenbin Jiang, Haoyi Zhang +2

School of Communication Engineering, Hangzhou Dianzi University, Hangzhou, China

Speech EnhancementSpiking Neural NetworksNeuromorphic Computing
AskPDF
Jun 21, 2026

Bridging Self-Supervised Learning and Speech Enhancement: A Wav2Vec2-Conditioned Framework

Diffusion models show potential for speech enhancement but lack linguistic guidance. We condition a diffusion-based model on wav2vec 2.0 features from noisy input, injected at the U-Net bottleneck via Feature-wise Linear Modulation (FiLM). Phonetic representations from wav2vec 2.0 features of degraded speech, anchor the reverse diffusion process. While a frozen wav2vec 2.0 encoder extracts features, a learned FiLM generator produces scale and shift parameters modulating the bottleneck with minimal overhead. Motivated by the optimal Bayesian causal estimator under a linear-Gaussian state-space model, FiLM coefficients are aggregated via exponential smoothing for temporal compression. Evaluation on VoiceBank-DEMAND and LibriMix shows competitive performance against the unconditioned baseline in PESQ, STOI, SI-SDR and DNSMOS. We consistently record an improvement of 0.4 on PESQ score, suggesting self-supervised representations effectively condition diffusion-based speech enhancement.

Shuubham Ojha, Carol Espy-Wilson

Institute for Systems Research, University of Maryland, College Park

Audio Diffusion ModelsSpeech EnhancementConditional Diffusion Models
AskPDF
Jun 19, 2026

Time-Frequency Weighted Losses for Phoneme Reconstruction in DNN-Based Speech Enhancement

Conventional training losses for speech enhancement based on the signal-to-distortion ratio (SDR) treat all time-frequency (TF) regions uniformly, overlooking the fine-grained spectral cues that are relevant to specific phoneme intelligibility. We propose a TF weighting framework that modulates the SDR objective based on local speech presence, speech-to-interference ratio (SIR), and spectral flux. By integrating these factors into a differentiable objective, the framework emphasizes TF bins with high speech-noise competition while also accounting for transient cues such as consonant bursts. Experimental results show that our approach improves objective frequency-weighted enhancement metrics, as well as phoneme recognition accuracy, particularly for consonants. Spectral analysis shows better reconstruction of mid-frequency structures at less adverse SIR.

Nasser-Eddine Monir, Paul Magron, Romain Serizel

Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France

Speech EnhancementAdaptive Loss Weighting
AskPDF
Jun 19, 2026

Post-Training Speech Enhancement Language Models with Perceptual Rewards

Speech enhancement language models achieve strong results when trained on discrete audio tokens, but their optimization relies on token-level cross-entropy rather than the perceptual metrics used for evaluation. We introduce a post-training stage for autoregressive speech enhancement language models using Group Sequence Policy Optimization (GSPO) with multi-metric perceptual rewards. Our method directly optimizes non-differentiable quality metrics (DNSMOS, WER, and UTMOS) as reward signals, without learned surrogates or offline preference pairs. Applied to two autoregressive base models, UniSE and GenSE, our approach achieves state-of-the-art results on the DNS2020 benchmark. A human evaluation ablation further shows that the composite multi-metric reward is preferred over any single-metric variant, confirming that multi-reward optimization avoids the reward hacking observed with single-metric training.

Frédéric Berdoz, Luca A. Lanzendörfer, Antonis Asonitis +1

ETH Zürich, Switzerland

Multi-Objective Reinforcement LearningRL for Language ModelsSpeech EnhancementSpeech Language ModelsLanguage Model Post-Training
AskPDF
Jun 18, 2026

Latency-Configurable Streaming Speech Enhancement via Asymmetric Temporal Padding

Streaming speech enhancement requires balancing algorithmic latency against quality, yet existing approaches largely treat this as a binary causal versus non-causal choice. LaCo-SENet addresses this issue with two mechanisms parameterized by a single training-time hyperparameter. First, asymmetric temporal padding redistributes past and future context in convolutions, enabling systematic latency configuration. Second, dual-buffer streaming combines state buffers for past context with lookahead buffers that supply future context at both the input and feature levels. Selective state updates also prevent future-frame leakage into the streaming state, ensuring training-inference consistency. On VoiceBank+DEMAND, a fixed-budget (1.37M parameters) backbone yields a family of models spanning 12.5-75.0 ms, with PESQ rising from 3.35 to 3.43. At just 12.5 ms (fully causal), a PESQ of 3.35 matches or exceeds the prior causal state-of-the-art (3.27 at 46.5 ms).

Yunsik Kim, Yoonyoung Chung

Department of Electrical Engineering, Pohang University of Science and Technology (POSTECH), Pohang 37673, Republic of Korea · Intus Co. Ltd., Pohang 37673, Republic of Korea

Speech EnhancementStreaming Algorithms
AskPDF
Jun 17, 2026

QC-GAN: A Parameter-Efficient Quaternion Conformer GAN for High-Fidelity Speech Enhancement

We propose a parameter-efficient speech enhancement framework, Quaternion Conformer GAN (QC-GAN), which combines a Quaternion Conformer generator with MetricGAN-based training. The Hamilton product encodes the magnitude and phase via structured weight sharing, reducing the number of layer parameters while preserving their interdependencies. A metric-learning discriminator was employed to maximize perceptual quality by optimizing the approximate perceptual evaluation scores. On the VoiceBank+DEMAND dataset, QC-GAN achieved a Perceptual Evaluation of Speech Quality (PESQ) score of 3.48 with only 0.89M parameters, delivering a performance comparable to state-of-the-art models at less than half their size. A 35K-parameter variant achieved a PESQ score of 3.23, surpassing conventional methods with significantly fewer parameters. Evaluation on the DNS-Challenge 3 dataset further confirmed generalization to real-world conditions.

Shogo Yamauchi, Hideaki Tamori, Makoto Sakai +2

The Asahi Shimbun Company, Japan · Tokyo Woman’s Christian University, Japan

TransformerComplex-Valued Neural NetworksSpeech EnhancementGenerative Adversarial NetworksConformer
AskPDF
Jun 16, 2026

Audio-visual Contrastive Alignment for Diffusion-based Visual-conditioned Speech Enhancement

Audio-visual speech enhancement (AVSE) exploits visual cues such as lip movements to recover speech in noisy environments. Recent work introduced diffusion-based unsupervised AVSE, where a speech diffusion model conditioned on visual features via cross-attention is trained and used as a data-driven prior for posterior sampling-based speech enhancement. Despite promising performance over its audio-only counterpart, the impact of explicitly enforcing cross-modal alignment in the fusion remains unclear. In this work, we propose to augment the diffusion training objective with a contrastive audio-visual loss to encourage stronger use of visual information while keeping the posterior sampling framework unchanged. Experiments across matched and mismatched test data show consistent improvements in interference suppression, signal reconstruction, and perceptual quality, with the largest gains at low SNRs. Code is available at https://github.com/ cexauce/AV-CA-DiffUSE

Colombe Mboungou, Mostafa Sadeghi, Jean-Eudes Ayilo +1

MULTISPEECH · Universit´e de Lorraine, CNRS, Inria, Loria, Nancy, France

Speech EnhancementAudio-Visual AlignmentNoise RobustnessDiffusion Models
AskPDF
Jun 15, 2026

Towards Robust Generative Speech Enhancement Using Vector Quantisation-Based Neural Audio Codec

This work investigates modelling strategies in continuous and discrete latent spaces in the vector quantisation (VQ)-based neural audio codec (NAC) speech enhancement (SE), along with the role of VQ regularisation. We propose cNAC-SE and dNAC-SE frameworks that predict continuous representations and discrete tokens in latent space, respectively. Theoretical analysis and visualisations in latent space are performed to exhibit their inherent modelling mechanisms. Experimental results show that the fully fine-tuned cNAC-SE model consistently outperforms all dNAC-SE variants across diverse test conditions and achieves leading performance among established generative approaches in DNS-MOS metrics. Comparison with the discriminative counterpart shows that VQ enhances robustness through an intrinsic effect of clean-prior-constrained regularisation, independent of discrete token processing. This highlights the transferable value of VQ regularisation to other continuous modelling methods.

Haixin Zhao, Nilesh Madhu

IDLab, Ghent University - imec, Belgium

Neural Audio CodecsVector QuantizationSpeech Enhancement
AskPDF
Jun 11, 2026

Generating Training Targets for Real-World Speech Enhancement via Close-to-Distant Microphone Projection

Training neural networks (NNs) for speech enhancement (SE) in distant speech-capturing scenarios requires paired distorted and clean reference speech signals. While such data are often generated through simulation, the mismatch between simulated and real recordings significantly limits SE accuracy. To address this issue, we propose Close-to-Distant microphone Projection (C2D projection), a method that generates paired data from real recordings captured by close and distant microphones. C2D projection estimates an optimal projection matrix that transforms close-microphone inputs into clean reference signals aligned with distant-microphone recordings, while simultaneously performing denoising. We show this projection can be effectively realized using a variant of the Parametric Multichannel Wiener Filter (PMWF). Experimental results demonstrate that an NN trained with C2D-projected data outperforms the state-of-the-art Guided Source Separation (GSS) on the challenging CHiME6 dinner party ASR task under oracle diarization, when using the enhanced output from GSS as an auxiliary input to the NN.

Tomohiro Nakatani, Rintaro Ikeshita, Naoyuki Kamo +2

NTT, Inc., Japan

Speech Enhancement
AskPDF
Jun 10, 2026

BASENet: Band-Adapted Speech Enhancement Network with Cross-Band Attention

Speech enhancement models typically apply uniform capacity across all frequencies, disregarding the non-uniform spectral resolution of human hearing. We propose BASENet, a frequency-adapted architecture that partitions the spectrum into Bark-scale bands and assigns each a scaled-capacity encoder derived from critical-band density, automatically granting deeper branches to perceptually dense low frequencies and lighter ones to high frequencies. A cross-band attention module captures harmonic dependencies across bands through compact frequency-pooled representations at linear complexity. Built on inverted residual blocks with dense connectivity and a convolutional recurrent network, BASENet achieves 3.55 PESQ and STOI96% on VoiceBank+DEMAND with only 0.83M parameters and 7.3 GMACs, the fewest parameters among all methods with PESQ > 3.50. A causal variant (3.44 PESQ) surpasses several non-causal baselines, confirming suitability for real-time streaming on resource-constrained devices.

Damien Martins Gomes, François Capman

Thales SIX GTS, FRANCE

Frequency-Domain Feature LearningSpeech Enhancement
AskPDF
Jun 7, 2026

G-MaP-SE: Guided Speech Enhancement via GMM-Based Prior Matching

Using speaker embeddings as conditioning can strengthen speech enhancement, but most methods either require clean enrollment audio or rely on embeddings extracted from noisy speech, which are fragile under noise and domain shift. We propose G-MaP-SE, a guided enhancement framework that builds a clean-speech embedding prior with a Gaussian Mixture Model (GMM) and refines a noisy conditioning embedding by matching it to this prior. The matched prior embedding is then injected into a time-frequency enhancement backbone via a lightweight gated fusion module. Experiments on VoiceBank+DEMAND and DNS Challenge 2020 datasets show that the proposed prior matching consistently outperforms noisy conditioning and substantially narrows the gap to an oracle clean-conditioning upper bound, while requiring no enrollment audio at inference time. The code, audio samples, and checkpoint are available.

Yike Zhu, Ziqian Wang, Zikai Liu +5

Speaker EmbeddingSpeech EnhancementNoise Robustness
AskPDF
Jun 4, 2026

DBHN-Net: Dual-Branch Hybrid Neural Network For Low-Complexity Monaural Speech Enhancement

Although artificial neural network (ANN) based speech enhancement (SE) methods demonstrate excellent performance, the high computational complexity and high energy consumption hinder their deployment in practical front-end processing tasks.} Currently, the spiking neural networks (SNNs) have shown potential in reducing power consumption. However, the discrete binary activation and complex spatio-temporal dynamics of SNNs often result in information loss. The current challenge therefore focuses on how to maintain performance and reduce computational complexity. To address this issue, this work propose a Dual-Branch Hybrid Neural (DBHN) Network. 1) In terms of network architecture: A dual-branch network integrating ANN and SNN was designed, where the SNN branch reduces power consumption while the ANN branch addresses information loss; The BandSplit and Time-Frequency (TF) -Mamba modules were developed to simultaneously compress energy consumption and enhance model performance; Spiking Feature Extraction Group (SFEG) and Information Transformation Block (ITB) components were implemented with residual connections to mitigate information loss while further refining feature representations. 2) To facilitate inter-branch information fusion: An Interaction module was designed to promote information exchange at various stages of the dual-branch network; A TF-Cross Attention-Fusion module was designed to perform time-frequency domain fusion of dual-branch information while data-adaptively guiding the SNN branch to retain more critical information. Results show that the proposed model maintains superior performance across three public datasets while achieving an average 7.5 fold reduction in computational complexity compared to baseline models.

Cunhang Fan, Enrui Liu, Jing Zhou +6

State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, (School of Computer Science and Technology), Anhui University, Hefei, 230601, Anhui, P. R. China · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · Institute of Acoustics, University of Chinese Academy of Sciences, Beijing 100190, China +1

Efficient Neural Network InferenceSpeech EnhancementSpiking Neural Networks
AskPDF
Jun 4, 2026

SB-RF: Schrödinger Bridge Rectified Flow for One-Step Robust Speech Enhancement

Generative models have shown promising results for speech enhancement (SE), but they often rely on multi-step inference, limiting low-latency deployment. We propose SB-RF, a one-step generative framework that integrates Rectified Flow (RF) with Schrödinger Bridge (SB) theory. During training, SB-RF samples intermediate states from an SB time marginal and trains a conditional velocity field with the RF velocity-matching objective. At inference, SB-RF starts from the noisy observation and applies a single Euler update. Experiments show that SB-RF achieves competitive performance among generative methods on the VoiceBank-DEMAND benchmark. To further assess performance beyond this standard setting, we evaluate SB-RF on a simulated low signal-to-noise ratio test set using an expanded training dataset. Under these conditions, SB-RF achieves superior performance over the compared baselines, supporting its potential for real-world applications.

Caixia Lu, Xueyang Lv, Penglong Hu +1

Rectified FlowSchrödinger BridgesOne-Step Generative ModelingSpeech EnhancementFlow-Based Generative ModelingNoise Robustness
AskPDF
Jun 2, 2026

Feasibility of Time-Domain DNN-Based Speech Enhancement on Embedded FPGA for Hearing Aid

Hearing aids impose strict latency and power constraints that current DNN-based speech enhancement systems struggle to meet on embedded hardware. We characterize this gap by deploying both speech separation and denoising using the lightweight SuDoRM-RF++ architecture on the AMD-Xilinx Kria KV260, evaluated at FP32 and 16-bit fixed-point precision for each task. Across these configurations, first-sample latency tracks with on-chip parameter caching rather than arithmetic throughput, identifying data movement as the primary bottleneck. Precision reduction halves the model memory footprint without compromising objective speech quality. The fixed-point denoising accelerator achieves a first-sample latency of 9.7ms, meeting the 10ms clinical threshold, while speech separation reaches 16.0~ms. These measurements establish concrete resource requirements for embedded DNN-based speech enhancement and quantify the remaining gap to hearing aid deployment.

Feyisayo Olalere, Umut Altin, Kiki van der Heijden +1

Radboud University, Donders Institute for Brain, Cognition, and Behaviour, The Netherlands · Mortimer B. Zuckerman Mind, Brain, Behavior Institute, Columbia University, USA.

Efficient Neural Network InferenceSpeech ProcessingSpeech EnhancementFPGA-Based Neural Network AccelerationSpeech SeparationOn-Device Inference
AskPDF
Jun 2, 2026

The Differentiable Auditory Loop (DAL): An ML Framework for Hyper-Personalized Hearing Aids

Conventional hearing aids rely on fixed, frequency-dependent amplification and compression to manage reduced sensitivity, which often fails to provide sufficient listening support in complex environments, such as situations with multiple speakers (the ``cocktail party'' problem). To more comprehensively address the underlying encoding dysfunctions of hearing loss, we introduce the Differentiable Auditory Loop (DAL), a new open-source framework for personalized hearing aid design and fitting. Our first implementation of DAL incorporates CARFAC, a differentiable model of human cochlear function, which we ported to JAX, to optimize a deep neural network to match impaired auditory neural activity patterns with a normal-hearing reference. To build a hearing aid with the fine-grained spectro-temporal signal processing required, we adopt SEANet, a waveform-to-waveform fully convolutional UNet generator. We fine-tune the network by comparing the outputs of a CARFAC model fitted to normal hearing with that of a CARFAC model fitted to match each subject's individual hearing impairment. The comparison is done using loss functions derived from the respective CARFAC neural activity pattern (NAP) outputs and stabilized auditory images (SAIs), the latter providing a 2D representation that captures phase-insensitive temporal structure in the auditory nerve output. Through gradient descent, the SEANet model learns to both denoise the input and compensate for the hearing loss modelled by the impaired CARFAC model. Across neural-representation and signal-fidelity metrics, the DAL-optimized SEANet model outperformed the tested master hearing aid (MHA) baselines. The DAL framework provides a practical path toward model-based, machine-learning-driven personalization of hearing aid signal processing. Next steps include hardware deployment to enable real-world clinical testing.

Alejandro Ballesta Rosen, Jason Mikiel-Hunter, Julian Maclaren +3

1Google Research Australia · 2Macquarie University

Speech Enhancement
AskPDF
Jun 2, 2026

Audio Spotforming via Post-Filtering Using Cross-Array Non-target Estimates

Audio spotforming is a technique for extracting target speech from noisy mixtures by utilizing multiple microphone arrays. Conventional methods estimate a shared target speech component from linearly separated signals obtained by each array using low-rank approximations and apply post filtering (PF) based on this estimated low-rank representation. However, owing to the mismatch between low-rank models and the complex structure of speech signals, directly relying on low-rank approximations for PF can degrade the speech extraction performance. In this study, we leverage the observation that non-target components located in the target speech direction from the perspective of one array can be spatially separated when viewed from other arrays. This insight motivates a new spotforming method for efficient post-filter estimation using non-target estimates across arrays instead of relying on low-rank approximations. Experiments demonstrate that the proposed method outperforms conventional spotforming methods.

Yuto Ishikawa, Li Li, Shogo Seki +1

CyberAgent, Inc., Japan · The University of Tokyo, Japan

Microphone Array ProcessingTarget Speaker ExtractionSpeech EnhancementSpatial Audio
AskPDF
Jun 1, 2026

A Comparison of Generative and Discriminative Methods for Speech Enhancement: Robustness, Complexity, and Hallucination

In this study, we conduct a comprehensive comparative analysis of generative and discriminative deep learning-based speech enhancement methods, specifically in noise reduction tasks. Our investigation focuses on evaluating their effectiveness under high and low signal-to-noise ratio conditions, considering both matched and mismatched training scenarios. We further investigate the impact of training data volume, model convergence speed, and interpret the performance differences in terms of objective results for the considered training paradigms. Additionally, we compare the complexity-performance trade-off and the practical viability of these approaches. To further strengthen the evaluation, we study the hallucination characteristics of generative approaches in terms of word error rate and phoneme similarity. The insights derived from this study provide empirical evidence to assist researchers and practitioners in understanding whether the perceptual gains of different approaches justify their computational cost in practical applications.

Shrishti Saha Shetu, Emanuël A. P. Habets, Andreas Brendel

International Audio Laboratories Erlangen, Am Wolfsmantel 33, 91058 Erlangen, Germany · Fraunhofer IIS, Am Wolfsmantel 33, 91058 Erlangen, Germany

Speech Enhancement
AskPDF