Recent studies have shown that post-deployment adaptation can improve the robustness of speech enhancement models in unseen noise conditions. However, existing methods often incur prohibitive computational and memory costs, limiting their suitability for on-device deployment. In this work, we investigate model adaptation in realistic settings with dynamic acoustic scene changes and propose a lightweight framework that augments a frozen backbone with low-rank adapters updated via self-supervised training. Experiments on sequential scene evaluations spanning 111 environments across 37 noise types and three signal-to-noise ratio ranges, including the challenging [-8, 0] dB range, show that our method updates fewer than 1% of the base model's parameters while achieving an average 1.51 dB SI-SDR improvement within only 20 updates per scene. Compared to state-of-the-art approaches, our framework achieves competitive or superior perceptual quality with smoother and more stable convergence, demonstrating its practicality for lightweight on-device adaptation of speech enhancement models under real-world acoustic conditions.
Figures & tables
Model
Sequential Scene?
Adaptation Framework
Adaptable Params
SNR ∈ [-8, 0] dB
SNR ∈ [0, 5] dB
SNR ∈ [5, 10] dB
#
%
PESQ
STOI
SI-SDR
PESQ
STOI
SI-SDR
PESQ
STOI
SI-SDR
Noisy Speech
1.08
67.88
-3.89
1.16
81.48
2.53
1.35
89.25
7.49
GRU*
Pretrained
1.16
71.01
3.86
1.35
84.85
6.82
1.57
90.99
9.58
✗
RemixIT
230,144
100
1.19
71.64
4.64
1.39
84.96
8.27
1.62
91.07
11.50
✗
Ours
512
0.22
1.22
71.96
4.71
1.44
85.34
8.36
1.67
91.38
11.43
✓
RemixIT
230,144
100
1.18
70.34
4.63
1.34
83.13
8.42
1.51
88.63
11.03
Table 1: Comparison of the proposed adaptation framework and RemixIT framework for GRU and DPRNN models in isolated (✗) and sequential (✓) scene settings. Bold values mark the best performance within each model configuration, while blue values indicate performance degradation relative to the pretrained baseline.
Figure 1: Per-update SNR improvement ( Δ SNR in dB) of adapted GRU and DPRNN backbones across three SNR ranges. Light curves indicate individual acoustic scenes. Adaptation with RemixIT achieves rapid early gains but exhibits unstable trajectories, while our method provides steady and consistent improvements over steps.
(Rank, Scale)
Adaptable Params
PESQ
STOI
SI-SDR
Noisy
-
1.20
79.54
2.04
Pretrained
-
1.36
82.28
6.75
(16, 1)
8,192
1.42
82.59
7.85
(32, 1)
16,384
1.43
82.88
7.91
(64, 1)
32,768
1.43
82.95
8.03
(1, 32)
512
1.42
82.81
8.04
Table 2: Performance of adapted GRU models with varying ranks and scaling factors. Results are averaged across all test scenes.
Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.
Test-time adaptation (TTA) offers a promising direction for improving speech enhancement models under mismatched acoustic conditions, without requiring access to labeled target data. In this work, we propose a single-utterance TTA method that regularizes a pretrained speech enhancement model using an autoregressive prior trained on clean speech latent representations extracted from a neural audio codec. Adaptation is performed by minimizing the Kullback-Leibler divergence between the enhanced speech distribution and the clean speech prior. Experiments across multiple noisy speech datasets show consistent improvements in speech quality, particularly under training-testing noise mismatch conditions. Code and audio examples are available online.
Sofiene Kammoun, Simon Leglaive, Xavier Alameda-Pineda +1
CentraleSup´elec, IETR (UMR CNRS 6164), France · Inria at Univ. Grenoble Alpes, CNRS, LJK, France · Signal Processing Group, University of Hamburg, Germany
State-of-the-art speech enhancement models benefit from large-scale labeled datasets, whereas singing voice separation models suffer from limited available training data. To address this limitation, we formulate singing voice separation as domain adaptation from speech enhancement to singing voice separation. We investigate two fine-tuning strategies: full fine-tuning and parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA) on a discriminative and a generative model. Models with either adaptation strategy outperform the same architectures trained from scratch by 0.29-1.8 dB in Signal-to-Distortion-Ratio. Full fine-tuning yields the highest singing voice separation performance, but catastrophic forgetting degrades speech enhancement performance. LoRA fine-tuning achieves competitive singing voice separation performance while preserving the original speech enhancement capability with only 6-12% additional parameters compared to the base speech enhancement model. Furthermore, the generative model shows improved generalization to an unseen test set. The results demonstrate that adapting pretrained speech enhancement models is an effective strategy for training singing voice separation models in data-scarce scenarios.
Paul A. Bereuter, Mark D. Plumbley, Alois Sontacchi
Institute of Electronic Music and Acoustics, University of Music and Performing Arts, Graz, Austria · Department of Informatics, King’s College London, London, United Kingdom
Speech enhancement (SE) models degrade under domain shifts and have to adapt to unseen target domains during deployment. Most existing test-time adaptation (TTA) methods for SE do so by adapting a subset of the model weights using a self-supervised loss, which requires backpropagation at test-time and permanently alters the model. We instead recalibrate the prediction itself and propose BLINC, a training-free TTA method that remaps the predicted time-frequency mask onto a bimodal target distribution by histogram matching. At test-time, the target distribution is parameterized from blind features of the noisy recording, so neither a reference distribution from a classical algorithm nor online metric optimization is involved. BLINC improves the overall quality of both evaluated SE models on almost every target condition and matches or exceeds the loss-based TTA baselines at minimal overhead.
Tobias Raichle, Ekaterina Gavrilko, Bin Yang
University of Stuttgart, Institute of Signal Processing and System Theory, Stuttgart, Germany