What Actually Makes Correlation-Based SSL Distillation Noise-Robust? A Mechanistic Correction

Fabian Ritter-Gutierrez; Nancy F. Chen; Eng Siong Chng

cs.SDOct 2, 2026

What Actually Makes Correlation-Based SSL Distillation Noise-Robust? A Mechanistic Correction

Authors: Fabian Ritter-Gutierrez, Nancy F. Chen, Eng Siong Chng

Organizations: Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), Singapore

Abstract

Self-supervised learning (SSL) speech models are accurate but large. Knowledge distillation compresses them, but the student loses the teacher's noise robustness. Correlation-based distillation addresses this with two terms: cross-correlation aligning student and teacher's representations, and self-correlation decorrelating the student's features. Both the original method and De'HuBERT credited the self-correlation term without isolating it. We show the opposite. On LibriSpeech-100 distillation with held-out CHiME-3 noise at 10,dB, an unbiased full-dimensional probe, a Pearson-variance decomposition, a same-noise causal control, and a per-dimension analysis identify the cross-correlation diagonal as the mechanism that encourages noise invariance, lowering noise-classification accuracy from 76.98% to 55.02%, whereas adding the self-correlation term back leaves it at 55.55%. The self-correlation term instead reorganises the feature space and improves accuracy on the clean downstream tasks, but removes essentially no noise. Across nine speech and music tasks this yields a concrete design rule for distillation: weight the cross-correlation term for noise robustness, tune the self-correlation term for better downstream performance, and sample teacher and student noise independently.

Figures & tables

Model	Loss	PCA-32 RF	LR-768
DistilHuBERT	$\mathcal{L}_{\text{KD}}$ (clean)	95.76	96.88
Setup 1	$\mathcal{L}_{\text{KD}}$	52.57	80.05
Setup 1	$\mathcal{L}_{\text{CL}}$	44.74	56.30
Setup 2	$\mathcal{L}_{\text{KD}}$	31.03	76.98
Setup 2	$\mathcal{L}_{\text{CL}}$	33.73	55.55
Setup 2	CC-only ( $\lambda_{\text{sc}}\!=\!0$ )	27.07	55.02

Table 1: Noise-classification accuracy (%) under two probes. PCA-32 RF: PCA to 32 dimensions then Random Forest (100 bootstrap seeds, mean). LR-768: logistic regression on all 768 dimensions. Lower is more noise-invariant.

Factor	LR-768 (%)	$\Delta$ (pp)	Interpretation
No noise aug. (baseline)	96.88	—	noise fully visible
$+$ Noise aug. ( $\mathcal{L}_{\text{KD}}$ )	76.98	$-$ 19.9	prerequisite (data)
$+$ CC matching ( $\lambda_{\text{sc}}\!=\!0$ )	55.02	$-$ 22.0	CC: dominant
$+$ SC term (full $\mathcal{L}_{\text{CL}}$ )	55.55	$+$ 0.53	SC: negligible

Table 2: Decomposition of noise suppression (Setup 2, LR-768 probe). Each row changes exactly one factor while holding the others constant. Values trace to Table 1 .

Model	Loss	Reverb	CHiME-3	FSD50K	Gaussian
DistilHuBERT	$\mathcal{L}_{\text{KD}}$ (clean)	100.0	97.2	90.4	100.0
Setup 1	$\mathcal{L}_{\text{KD}}$	92.9	78.7	66.1	83.0
Setup 1	$\mathcal{L}_{\text{CL}}$	80.4	54.6	41.3	48.7
Setup 2	$\mathcal{L}_{\text{KD}}$	89.3	75.0	65.5	78.2
Setup 2	$\mathcal{L}_{\text{CL}}$	80.9	65.9	34.3	43.1
Setup 2	CC-only ( $\lambda_{\text{sc}}\!=\!0$ )	75.2	58.4	42.6	43.5

Table 3: Per-class noise-classification accuracy (%, per-class recall) under the LR-768 probe. Chance is 25%.

Baselines
			ASR (WER $\downarrow$ )			KS (Acc $\uparrow$ )			IC (Acc $\uparrow$ )			ER (Acc $\uparrow$ )
Upstream	Dist.	Loss	cln	nsy	$\Delta$	cln	nsy	$\Delta$	cln	nsy	$\Delta$	cln	nsy	$\Delta$
HuBERT Base	–	–	7.84	10.4	2.56	96.30	93.74	2.56	98.34	90.96	7.38	64.92	57.69	7.23
HuBERT+ Base	–	–	8.08	9.41	1.33	96.17	94.94	1.23	98.37	97.02	1.35	66.07	63.25	2.82
DistilHuBERT	–	$\mathcal{L}_{\text{KD}}$	15.83	29.65	13.82	95.12	90.29	4.83	91.20	66.71	24.49	61.49	52.17	9.32
Noise-robust distillation

Table 4: Speech-task results from [ 13 ] under clean and noisy (CHiME-3, 10 dB SNR) conditions. HuBERT/HuBERT+ are 95M parameters; distilled students are 23M. The lower block uses the HuBERT teacher under independent noise (Setup 2). Metrics: WER%

\downarrow

for ASR, Acc%

\uparrow

otherwise.

\Delta

is clean-to-noisy degradation (smaller is more robust). Shaded row uses

\mathcal{L}_{\text{CL}}

			SingerID			VocID			InstCls			ESC-50
Upstream	Dist.	Loss	cln	nsy	$\Delta$	cln	nsy	$\Delta$	cln	nsy	$\Delta$	cln	nsy	$\Delta$
DistilHuBERT	–	$\mathcal{L}_{\text{KD}}$	68.04	36.41	31.63	62.19	55.88	6.31	61.42	22.75	38.67	68.24	52.30	15.94
DistilHuBERT	Both	$\mathcal{L}_{\text{KD}}$	57.84	42.21	15.63	59.79	56.68	3.11	60.64	33.45	27.19	67.30	59.40	7.90
DistilHuBERT	Both	$\mathcal{L}_{\text{CL}}$	54.04	41.46	12.58	62.90	58.43	4.47	60.01	37.85	22.16	70.30	64.69	5.61

Table 5: Cross-domain music and environmental-sound results under clean and noisy (CHiME-3, 10 dB SNR) conditions. All metrics Acc%

\uparrow

;

\Delta=

clean

-

noisy. The first row is vanilla DistilHuBERT without noise augmentation; the lower block is the HuBERT teacher under independent noise (Setup 2). Shaded row uses

\mathcal{L}_{\text{CL}}

. GenreID is omitted because the GTZAN benchmark is known to be unreliable.

Model	Noise sampling	LR-768 (%)
DistilHuBERT (no noise aug.)	N/A	96.88
Setup 2 $\mathcal{L}_{\text{CL}}$	Same	95.38
Setup 2 $\mathcal{L}_{\text{KD}}$	Independent	76.98
Setup 2 $\mathcal{L}_{\text{CL}}$	Independent	55.55

Table 6: Same-noise control. LR-768 noise-classification accuracy (%). Chance is 25%.

Model	Noise	LR-768 (%)	Max (%)	Mean (%)	Dims $>$ 30%
DistilHuBERT (no aug.)	N/A	96.88	–	–	727
$\mathcal{L}_{\text{CL}}$	Same	95.4	58.1	39.6	719
$\mathcal{L}_{\text{KD}}$	Indep.	77.0	31.4	26.6	16
$\mathcal{L}_{\text{CL}}$	Indep.	55.5	35.2	26.9	34
CC-only ( $\lambda_{\text{sc}}\!=\!0$ )	Indep.	55.0	30.5	26.3	5

Table 7: Per-dimension noise analysis. Rows are Setup 2 models plus a no-augmentation reference (first row). LR-768 is total noise-classification accuracy. “Dims

>

30%” counts the 768 feature dimensions individually above the 30% threshold.

Figure 1: Per-dimension noise-classification accuracy on the 768 feature dimensions, ranked from most to least noise-informative (chance 25%). At matched total noise near 55%, the full loss $\mathcal{L}_{\text{CL}}$ spreads noise over many more individually informative dimensions than the cross-correlation-only model, while carrying the same total. Identical noise saturates almost every dimension.

	IC (Acc $\uparrow$ )		KS (Acc $\uparrow$ )		ASR (WER $\downarrow$ )		ER (Acc $\uparrow$ )
Loss	cln	nsy	cln	nsy	cln	nsy	cln	nsy
$\mathcal{L}_{\text{KD}}$	93.03	89.29	95.41	93.15	16.98	20.47	60.73	59.03
CC-only ( $\lambda_{\text{sc}}\!=\!0$ )	95.60	90.83	95.16	94.31	16.40	20.83	62.13	60.38
Full $\mathcal{L}_{\text{CL}}$	96.61	94.06	95.59	94.38	15.35	18.18	64.71	63.09

Table 8: Self-correlation ablation (Setup 2, HuBERT teacher). CC-only sets

\lambda_{\text{sc}}\!=\!0

; full

\mathcal{L}_{\text{CL}}

uses

\lambda_{\text{sc}}\!=\!5\times 10^{-6}

. cln/nsy = clean/noisy.

Self-Supervised Knowledge Distillation

Self-Supervised Knowledge Distillation transfers useful model behavior or representations between teacher and student without labeled targets.

See all 22 papers

Oct 2, 2026

Correlation-Based Distillation Yields More Mergeable Speech-Music Encoders

A single compact encoder for both speech and music removes the need to maintain a separate model per domain. A practical recipe distils a speech teacher and a music teacher into two small students, then merges them into one model. Two merging algorithms exist: interpolating the student weights from a shared initialisation, or permuting their channels into alignment before averaging. Prior work fixed the distillation loss in both, leaving open whether the loss itself affects how well the students merge. We show that it does. Holding architecture and evaluation fixed, we replace the DistilHuBERT loss \Lkd{} with a correlation-based loss \Lcl{}. Under weight interpolation, \Lcl{} students stay closer to their own better endpoint in

17

18

comparisons and lead on the speech tasks at every interpolated weight. Under activation permutation, they need fewer channels rearranged at three layers, their matched channels correlate more strongly, and they again lead on speech. The two merging algorithms share no mechanism, yet both improve under the same loss substitution. This suggests that the gain lies in the representation \Lcl{} produces rather than in either algorithm.

Fabian Ritter-Gutierrez

Nanyang Technological University, Singapore

Audio Representation Learning Knowledge Distillation Model Merging

Ask PDF

Mar 31, 2026

HARNESS: Lightweight Distilled Arabic Speech Foundation Models

Large self-supervised speech (SSL) models achieve strong downstream performance, but their size limits deployment in resource-constrained settings. We present HArnESS, an Arabic-centric self-supervised speech model family trained from scratch with iterative self-distillation, together with lightweight student variants that offer strong accuracy-efficiency trade-offs on Automatic Speech Recognition (ASR), Dialect Identification (DID), and Speech Emotion Recognition (SER). Our approach begins with a large bilingual Arabic-English teacher and progressively distills its knowledge into compressed student models while preserving Arabic-relevant acoustic and paralinguistic representations. We further study PCA-based compression of the teacher supervision signal to better match the capacity of shallow and thin students. Compared with HuBERT and XLS-R, HArnESS consistently improves performance on Arabic downstream tasks, while the compressed models remain competitive under substantial structural reduction. These results position HArnESS as a practical and accessible Arabic-centric SSL foundation for real-world speech applications.

Vrunda N. Sukhadia, Shammur Absar Chowdhury

Amazon India · Qatar Computing Research Institute, HBKU, Qatar

Speech Foundation Models Speech Processing Self-Supervised Knowledge Distillation Automatic Speech Recognition

Ask PDF

Sep 16, 2026

Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding

Self-supervised learning (SSL) has improved speech representations, yet performance degrades in pathological domains such as electrolaryngeal (EL) speech, and the computational footprint of SSL models limits their applicability in real-time, on-device deployment. We propose a multi-teacher knowledge distillation framework to train a lightweight, streaming content encoder that generalizes across healthy (HE) and EL speech. Two teachers are distilled progressively: a frozen SSL model providing discrete phonetic cluster targets from HE speech, and an EL-fine-tuned speech recognition model supplying continuous bottleneck feature targets. Evaluated via downstream speech recognition, our approach reduces the EL word error rate to 21.2%, compared to 39.3% for the strongest zero-shot SSL baseline. Among causal convolutional, Transformer, Conformer, and Mamba-based student architectures, a Mel-Conformer achieves the best combination of EL accuracy and computational efficiency. The final encoder contains 21.9,M parameters and runs at a real-time factor of 0.30 under ONNX Runtime on a single CPU core.

Benedikt Mayrhofer, Enrique Orozco Olivares, Franz Pernkopf +2

Signal Processing and Speech Communication Laboratory, Graz University of Technology · Comprehensive Center for AI in Medicine, Medical University of Vienna · Dept. of Otolaryngology, Head & Neck Surgery, Div. of Phoniatrics-Logopedics, Speech & Hearing Sci. Lab, Med. Univ. of Vienna

Audio Representation Learning Streaming ASR Multi-Teacher Knowledge Distillation

Ask PDF

Abstract

Self-Supervised Knowledge Distillation

Explore similar work

Correlation-Based Distillation Yields More Mergeable Speech-Music Encoders

HARNESS: Lightweight Distilled Arabic Speech Foundation Models

Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding