Self-supervised learning (SSL) representations are widely used in speech restoration (SR), and the prevailing trend is SSL feature cleansing: fine-tuning the SSL to recover clean features from degraded speech, then conditioning the restoration model on them. However, the representational capability of SSL when encoding degraded speech has been underexplored, and we ask whether cleansing always benefits SR training. Across SSL encoders and perceptual quality proxies, we discover perceptual correlation collapse---under severe degradation, cleansing degraded SSL features toward clean ones misaligns with the SR objective. We hypothesize the restoration backbone can supply the SSL with clues to resolve this collapse, since its intermediate layers already produce partially restored features. We therefore propose Early Restoration Injection (ERI), which routes intermediate features from the restoration backbone into the SSL through a lightweight conditioning network. ERI excels over the unconditioned baseline and, on the VoiceFixer benchmark, achieves competitive performance among multiple baselines.
Figures & tables
Figure 1: t-SNE plots of speech degraded with additive noise at varying SNR level, visualized in WavLM feature space.
Figure 2: SSL-wise plot of perceptual correlation.
Figure 3: Model architecture and training framework.
Model
# Params (M) total / trainable
OVRL ↑
UTMOS ↑
PESQ ↑
ESTOI ↑
LPS ↑
Degraded
-
2.39
2.63
1.94
0.58
0.84
VoiceFixer
112 / 112
3.00
3.44
2.04
0.56
0.80
TF-GridNet
14.4 / 14.4
3.06
3.41
2.47
0.68
0.88
AnyEnhance
364 / 324
3.14
3.75
2.45
0.67
0.88
SIDON
250 / 57.4
3.20
3.97
2.05
0.58
0.88
SEMamba++
2.7 / 2.7
3.10
3.75
2.76
0.70
0.87
Table 1: Performance evaluation on the VoiceFixer testset.
ERI
SSL
Stage 1 steps (k)
SCOREQ↑
UTMOS↑
PESQ↑
LPS↑
✓
WavLM
500 (Full)
3.97
4.06
2.50
0.90
✗
3.93
4.03
2.42
0.88
✓
WavLM
300
3.95
4.04
2.41
0.89
✗
3.88
3.99
2.41
0.88
✓
HuBERT
500
3.93
4.03
2.41
0.87
✗
3.86
3.96
2.33
0.83
Table 2: Performance comparison on ERI.
Design
Params
SCOREQ↑
UTMOS↑
PESQ↑
LPS↑
Ours (default)
24M
3.95
4.04
2.41
0.89
ERI layer 2 → 1
24M
3.91
4.03
2.42
0.88
w/o sg on SSL cond. fus.
24M
3.92
4.02
2.42
0.88
AdaLN-based ERI
32M
3.94
4.04
2.43
0.89
Table 3: Design study. We used 300k steps for stage 1 training for this study.
Speech self-supervised learning aims to learn general-purpose representations for downstream speech tasks. However, current approaches rely on complex, carefully designed prediction targets. We challenge this necessity with GLaS-JEPA, a framework that directly predicts the current encoder's continuous representations at masked positions, without contrastive learning, discrete targets, or separate EMA target encoders. We prevent representation collapse using SIGReg representation-space regularization, eliminating the need for engineered target-generation mechanisms. Pretrained on 960 hours of LibriSpeech, our 57M-parameter model achieves a 6.89% WER on frozen-encoder SUPERB ASR and a 25.87% CER on slot filling, outperforming the best non-distilled sub-90M baselines by 43.1% and 22.0%, respectively. These results demonstrate that highly competitive speech representations can emerge from a radically simplified training recipe.
Gaspard Botté, Séverin Baroudi, Samir Sadok +5
Concordia University · Mila – Qu´ebec AI Institute · Univ Toulon, Aix Marseille Univ, CNRS, LIS +4
We propose Online Latent prediction with Invariant Views and rEconstruction (OLIVE), a self-supervised speech representation learning framework that jointly optimizes analysis and synthesis objectives. OLIVE combines view-augmented masked latent prediction with waveform reconstruction under a unified objective. Reconstruction constrains early encoder features to retain signal-level information, while masked latent prediction shapes later contextual representations toward invariance for robust downstream performance. We show that these objectives enable representations that support a broad range of tasks. In particular, OLIVE improves results on generation and speaker tasks, maintains competitive performance on recognition and semantic tasks, and improves waveform reconstruction.
Karl El Hajal, Mathew Magimai. -Doss
Idiap Research Institute, Switzerland · EPFL, Switzerland
Unsupervised speech denoising methods such as MixIT and noisy-target training can be trained directly on noisy recordings, avoiding the sim-to-real gap of synthetic training data, but remain well below supervised performance. Teacher--student self-training, as in RemixIT, narrows this gap by remixing a teacher's estimates into new mixtures that supervise a student; however, since the student is trained only toward the teacher's estimates, teacher errors, in particular over-suppressed speech, are never recovered and propagate as the teacher is updated from the student. We propose SNOAR, a self-training framework that corrects the teacher's estimates before use: the noise leaked into its speech estimate is removed by differential noise filtering (DNF), and part of the noisy observation is added back in to restore suppressed speech and attenuate artifacts. We show that the two corrections depend on each other, as without DNF the loop collapses to the identity mapping. Compared to RemixIT, SNOAR is more stable along training and outperforms it in signal-based, perceptual, and word error rate metrics, reaching 97% of the SI-SDR improvement of a fully-supervised topline on WHAM!; on real CHiME-3 recordings, it improves every metric of three different teachers, while RemixIT degrades their perceptual quality.
Samuele Cornell, Matthew Maciejewski
Carnegie Mellon University, USA · Johns Hopkins University, USA