Domain-Incremental Learning for Generative Speech Enhancement

Manjunath Mulimani; Annamaria Mesaros; Minje Kim; Jesper Rindom Jensen

eess.ASSep 28, 2026

Domain-Incremental Learning for Generative Speech Enhancement

Authors: Manjunath Mulimani, Annamaria Mesaros, Minje Kim, Jesper Rindom Jensen

Organizations: Aalborg University, Department of Electronic Systems, Denmark · Tampere University, Signal Processing Research Centre, Finland · University of Illinois Urbana-Champaign, Siebel School of Computing and Data Science, USA

Abstract

We propose a domain-incremental learning framework for generative speech enhancement (SE) that learns from a sequence of datasets or domains recorded under diverse acoustic conditions. Fine-tuning a pretrained model on continuously evolving domains leads to catastrophic forgetting of previously acquired knowledge, while zero-shot generalization often fails to adequately adapt to unseen domains. To address these challenges, we first develop a novel language model-based generative SE model that we then use as a pretrained backbone and incrementally adapt it to acoustically mismatched domains using lightweight domain-specific Low-Rank Adaptation. The proposed framework enables the model to acquire enhancement capabilities for new domains while preserving performance on previously learned domains. Evaluated on four heterogeneous speech datasets, our approach effectively adapts to new domains without forgetting previously learned domains.

Figures & tables

Panel of Figure 1 — Figure 1: An overview of the proposed DIL for generative speech enhancement. (a) The base model $\mathcal{M}$ is trained on domain $\mathcal{D}_{0}$ . (b) LoRA is added to $\mathcal{M}$ for each incremental domain $\mathcal{D}_{t}$ .

Method	$\mathcal{D}_{1}$ DNS	$\mathcal{D}_{2}$ EARS	$\mathcal{D}_{3}$ LibriTTS-R	$\mathcal{D}_{4}$ VoiceBank
GRL	0.89	0.82	0.84	0.83
FT	0.85	0.78	0.74	0.73
Joint FT	0.85	0.80	0.82	0.83
DIL-GenSE	0.90	0.87	0.88	0.88

Table 1: Average

\text{SBS}_{t}

(

\uparrow

) across the current domain

\mathcal{D}_{t}

and all previously seen domains

\mathcal{D}_{1,\ldots,t-1}

under the domain-aware setup.

Panel of Figure 2 — Figure 2: Comparison of the proposed DIL-GenSE with zero-shot generalization and fine-tuning. (a) SBS ( $\uparrow$ ) of the generalization and the DIL-GenSE method in the current domain $\mathcal{D}_{t}$ . (b) SBS ( $\uparrow$ ) at the $\mathcal{D}_{t}$ and average forgetting $\text{FR}_{t}$ over the previously encountered domains $\mathcal{D}_{1,\ldots,t-1}$ learned for the FT.

$\mathcal{D}_{1}$ DNS	$\mathcal{D}_{2}$ EARS	$\mathcal{D}_{3}$ LibriTTS-R	$\mathcal{D}_{4}$ VoiceBank
0.90	0.79	0.80	0.76

Table 2: Average

\text{SBS}_{t}

(

\uparrow

) across the current domain

\mathcal{D}_{t}

and all previously seen domains

\mathcal{D}_{1,\ldots,t-1}

under the domain-agnostic setup.

	DNSMOS $\uparrow$
Method	SIG	BAK	OVL	SBS $\uparrow$
GenSE [ 31 ]	2.60	3.22	2.14	0.52
DIL-GenSE Agnostic	3.39	3.99	3.16	0.74
DIL-GenSE Aware	3.56	4.05	3.31	0.84

Table 3: Comparison of proposed DIL-GenSE with GenSE [ 31 ] on the EARS domain.

Explore similar work

Teaching Speech Enhancement Models to Sing: Domain Adaptation from Speech Enhancement to Singing Voice Separation
Jul 13, 2026Paul A. Bereuter, Mark D. Plumbley, Alois SontacchiSpeech Enhancement Speech Separation
Institute of Electronic Music and Acoustics, University of Music and Performing Arts, Graz, Austria · Department of Informatics, King’s College London, London, United Kingdom
DriftSE: Speech Enhancement with Generative Drifting
Sep 14, 2026Liang Xu, Diego Caviedes-Nozal, W. Bastiaan Kleijn +2Speech Enhancement Acoustic Latent Space
Victoria University of Wellington, New Zealand · GN Advanced Science, Denmark · Lincoln University
Transcript-Supervised Post-Training of Generative Speech Enhancement on Real Recordings via Reinforce Adjoint Matching
Sep 24, 2026Julius Richter, Christoph Boeddeker, Yoshiki Masuyama +4Speech Enhancement Post-Training
Mitsubishi Electric Research Laboratories (MERL), USA · Mitsubishi Electric Corporation, Japan

Jul 13, 2026, cs.SDJ/K move · Enter open · S save

Teaching Speech Enhancement Models to Sing: Domain Adaptation from Speech Enhancement to Singing Voice Separation

Paul A. Bereuter, Mark D. Plumbley, Alois Sontacchi

Institute of Electronic Music and Acoustics, University of Music and Performing Arts, Graz, Austria · Department of Informatics, King’s College London, London, United Kingdom

Speech Enhancement Speech Separation Domain Adaptation

Open paper AskPDF arXiv

State-of-the-art speech enhancement models benefit from large-scale labeled datasets, whereas singing voice separation models suffer from limited available training data. To address this limitation, we formulate singing voice separation as domain adaptation from speech enhancement to singing voice separation. We investigate two fine-tuning strategies: full fine-tuning and parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA) on a discriminative and a generative model. Models with either adaptation strategy outperform the same architectures trained from scratch by 0.29-1.8 dB in Signal-to-Distortion-Ratio. Full fine-tuning yields the highest singing voice separation performance, but catastrophic forgetting degrades speech enhancement performance. LoRA fine-tuning achieves competitive singing voice separation performance while preserving the original speech enhancement capability with only 6-12% additional parameters compared to the base speech enhancement model. Furthermore, the generative model shows improved generalization to an unseen test set. The results demonstrate that adapting pretrained speech enhancement models is an effective strategy for training singing voice separation models in data-scarce scenarios.

Abstract

Explore similar work

Teaching Speech Enhancement Models to Sing: Domain Adaptation from Speech Enhancement to Singing Voice Separation

DriftSE: Speech Enhancement with Generative Drifting

Transcript-Supervised Post-Training of Generative Speech Enhancement on Real Recordings via Reinforce Adjoint Matching