When Do Self-Supervised Representations Help Speech Restoration?

Yongjoon Lee; Younghoo Kwon; Jung-Woo Choi

cs.SDOct 22, 2026

When Do Self-Supervised Representations Help Speech Restoration?

Authors: Yongjoon Lee, Younghoo Kwon, Jung-Woo Choi

Organizations: School of Electrical Engineering, KAIST, Daejeon, South Korea · Department of AI Systems, KAIST, Daejeon, South Korea

Abstract

Self-supervised learning (SSL) representations are widely used in speech restoration (SR), and the prevailing trend is SSL feature cleansing: fine-tuning the SSL to recover clean features from degraded speech, then conditioning the restoration model on them. However, the representational capability of SSL when encoding degraded speech has been underexplored, and we ask whether cleansing always benefits SR training. Across SSL encoders and perceptual quality proxies, we discover perceptual correlation collapse---under severe degradation, cleansing degraded SSL features toward clean ones misaligns with the SR objective. We hypothesize the restoration backbone can supply the SSL with clues to resolve this collapse, since its intermediate layers already produce partially restored features. We therefore propose Early Restoration Injection (ERI), which routes intermediate features from the restoration backbone into the SSL through a lightweight conditioning network. ERI excels over the unconditioned baseline and, on the VoiceFixer benchmark, achieves competitive performance among multiple baselines.

Figures & tables

Figure 1: t-SNE plots of speech degraded with additive noise at varying SNR level, visualized in WavLM feature space.

Panel of Figure 2 — Figure 2: SSL-wise plot of perceptual correlation.

Figure 3: Model architecture and training framework.

Model	# Params (M) total / trainable	OVRL ↑	UTMOS ↑	PESQ ↑	ESTOI ↑	LPS ↑
Degraded	-	2.39	2.63	1.94	0.58	0.84
VoiceFixer	112 / 112	3.00	3.44	2.04	0.56	0.80
TF-GridNet	14.4 / 14.4	3.06	3.41	2.47	0.68	0.88
AnyEnhance	364 / 324	3.14	3.75	2.45	0.67	0.88
SIDON	250 / 57.4	3.20	3.97	2.05	0.58	0.88
SEMamba++	2.7 / 2.7	3.10	3.75	2.76	0.70	0.87

Table 1: Performance evaluation on the VoiceFixer testset.

ERI	SSL	Stage 1 steps (k)	$\text{SCOREQ}^{\uparrow}$	$\text{UTMOS}^{\uparrow}$	$\text{PESQ}^{\uparrow}$	$\text{LPS}^{\uparrow}$
✓	WavLM	500 (Full)	3.97	4.06	2.50	0.90
✗	WavLM	500 (Full)	3.93	4.03	2.42	0.88
✓	WavLM	300	3.95	4.04	2.41	0.89
✗	WavLM	300	3.88	3.99	2.41	0.88
✓	HuBERT	500	3.93	4.03	2.41	0.87
✗	HuBERT	500	3.86	3.96	2.33	0.83

Table 2: Performance comparison on ERI.

Design	Params	$\text{SCOREQ}^{\uparrow}$	$\text{UTMOS}^{\uparrow}$	$\text{PESQ}^{\uparrow}$	$\text{LPS}^{\uparrow}$
Ours (default)	24M	3.95	4.04	2.41	0.89
ERI layer 2 $\rightarrow$ 1	24M	3.91	4.03	2.42	0.88
w/o sg on SSL cond. fus.	24M	3.92	4.02	2.42	0.88
AdaLN-based ERI	32M	3.94	4.04	2.43	0.89

Table 3: Design study. We used 300k steps for stage 1 training for this study.

Figure 4: Perceptual gain ratio and SCOREQ value.

Self-Supervised Learning

Self-supervised learning uses structure within unlabeled data to train representations and models for downstream tasks.

See all 256 papers

GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets
Sep 29, 2026Gaspard Botté, Séverin Baroudi, Samir Sadok +5Joint-Embedding Predictive Architecture Predictive Representation Learning
Concordia University · Mila – Qu´ebec AI Institute · Univ Toulon, Aix Marseille Univ, CNRS, LIS +4
OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL
Jun 29, 2026Karl El Hajal, Mathew Magimai. -DossPredictive Representation Learning Self-Supervised Speech Representation Learning
Idiap Research Institute, Switzerland · EPFL, Switzerland
SNOAR: Self-training with Noise filtering, Observation adding, and Remixing for Speech Denoising Without Clean References
Oct 14, 2026Samuele Cornell, Matthew MaciejewskiSpeech Enhancement
Carnegie Mellon University, USA · Johns Hopkins University, USA

Sep 29, 2026J/K move · Enter open · S save

GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets

Gaspard Botté, Séverin Baroudi, Samir Sadok, Francesco Paissan+4

Concordia University · Mila – Qu´ebec AI Institute · Univ Toulon, Aix Marseille Univ, CNRS, LIS+4

Joint-Embedding Predictive Architecture Predictive Representation Learning Self-Supervised Speech Representation Learning

Open paper AskPDF arXiv

Speech self-supervised learning aims to learn general-purpose representations for downstream speech tasks. However, current approaches rely on complex, carefully designed prediction targets. We challenge this necessity with GLaS-JEPA, a framework that directly predicts the current encoder's continuous representations at masked positions, without contrastive learning, discrete targets, or separate EMA target encoders. We prevent representation collapse using SIGReg representation-space regularization, eliminating the need for engineered target-generation mechanisms. Pretrained on 960 hours of LibriSpeech, our 57M-parameter model achieves a 6.89% WER on frozen-encoder SUPERB ASR and a 25.87% CER on slot filling, outperforming the best non-distilled sub-90M baselines by 43.1% and 22.0%, respectively. These results demonstrate that highly competitive speech representations can emerge from a radically simplified training recipe.

Abstract

Self-Supervised Learning

Explore similar work

GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets

OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL

SNOAR: Self-training with Noise filtering, Observation adding, and Remixing for Speech Denoising Without Clean References