What Actually Makes Correlation-Based SSL Distillation Noise-Robust? A Mechanistic Correction

Fabian Ritter-Gutierrez; Nancy F. Chen; Eng Siong Chng

cs.SDOct 2, 2026

What Actually Makes Correlation-Based SSL Distillation Noise-Robust? A Mechanistic Correction

Authors: Fabian Ritter-Gutierrez, Nancy F. Chen, Eng Siong Chng

Organizations: Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), Singapore

Abstract

Self-supervised learning (SSL) speech models are accurate but large. Knowledge distillation compresses them, but the student loses the teacher's noise robustness. Correlation-based distillation addresses this with two terms: cross-correlation aligning student and teacher's representations, and self-correlation decorrelating the student's features. Both the original method and De'HuBERT credited the self-correlation term without isolating it. We show the opposite. On LibriSpeech-100 distillation with held-out CHiME-3 noise at 10,dB, an unbiased full-dimensional probe, a Pearson-variance decomposition, a same-noise causal control, and a per-dimension analysis identify the cross-correlation diagonal as the mechanism that encourages noise invariance, lowering noise-classification accuracy from 76.98% to 55.02%, whereas adding the self-correlation term back leaves it at 55.55%. The self-correlation term instead reorganises the feature space and improves accuracy on the clean downstream tasks, but removes essentially no noise. Across nine speech and music tasks this yields a concrete design rule for distillation: weight the cross-correlation term for noise robustness, tune the self-correlation term for better downstream performance, and sample teacher and student noise independently.

Figures & tables

Model	Loss	PCA-32 RF	LR-768
DistilHuBERT	$\mathcal{L}_{\text{KD}}$ (clean)	95.76	96.88
Setup 1	$\mathcal{L}_{\text{KD}}$	52.57	80.05
Setup 1	$\mathcal{L}_{\text{CL}}$	44.74	56.30
Setup 2	$\mathcal{L}_{\text{KD}}$	31.03	76.98
Setup 2	$\mathcal{L}_{\text{CL}}$	33.73	55.55
Setup 2	CC-only ( $\lambda_{\text{sc}}\!=\!0$ )	27.07	55.02

Table 1: Noise-classification accuracy (%) under two probes. PCA-32 RF: PCA to 32 dimensions then Random Forest (100 bootstrap seeds, mean). LR-768: logistic regression on all 768 dimensions. Lower is more noise-invariant.

Factor	LR-768 (%)	$\Delta$ (pp)	Interpretation
No noise aug. (baseline)	96.88	—	noise fully visible
$+$ Noise aug. ( $\mathcal{L}_{\text{KD}}$ )	76.98	$-$ 19.9	prerequisite (data)
$+$ CC matching ( $\lambda_{\text{sc}}\!=\!0$ )	55.02	$-$ 22.0	CC: dominant
$+$ SC term (full $\mathcal{L}_{\text{CL}}$ )	55.55	$+$ 0.53	SC: negligible

Table 2: Decomposition of noise suppression (Setup 2, LR-768 probe). Each row changes exactly one factor while holding the others constant. Values trace to Table 1 .

Model	Loss	Reverb	CHiME-3	FSD50K	Gaussian
DistilHuBERT	$\mathcal{L}_{\text{KD}}$ (clean)	100.0	97.2	90.4	100.0
Setup 1	$\mathcal{L}_{\text{KD}}$	92.9	78.7	66.1	83.0
Setup 1	$\mathcal{L}_{\text{CL}}$	80.4	54.6	41.3	48.7
Setup 2	$\mathcal{L}_{\text{KD}}$	89.3	75.0	65.5	78.2
Setup 2	$\mathcal{L}_{\text{CL}}$	80.9	65.9	34.3	43.1
Setup 2	CC-only ( $\lambda_{\text{sc}}\!=\!0$ )	75.2	58.4	42.6	43.5

Table 3: Per-class noise-classification accuracy (%, per-class recall) under the LR-768 probe. Chance is 25%.

Baselines
			ASR (WER $\downarrow$ )			KS (Acc $\uparrow$ )			IC (Acc $\uparrow$ )			ER (Acc $\uparrow$ )
Upstream	Dist.	Loss	cln	nsy	$\Delta$	cln	nsy	$\Delta$	cln	nsy	$\Delta$	cln	nsy	$\Delta$
HuBERT Base	–	–	7.84	10.4	2.56	96.30	93.74	2.56	98.34	90.96	7.38	64.92	57.69	7.23
HuBERT+ Base	–	–	8.08	9.41	1.33	96.17	94.94	1.23	98.37	97.02	1.35	66.07	63.25	2.82
DistilHuBERT	–	$\mathcal{L}_{\text{KD}}$	15.83	29.65	13.82	95.12	90.29	4.83	91.20	66.71	24.49	61.49	52.17	9.32
Noise-robust distillation

Table 4: Speech-task results from [ 13 ] under clean and noisy (CHiME-3, 10 dB SNR) conditions. HuBERT/HuBERT+ are 95M parameters; distilled students are 23M. The lower block uses the HuBERT teacher under independent noise (Setup 2). Metrics: WER%

\downarrow

for ASR, Acc%

\uparrow

otherwise.

\Delta

is clean-to-noisy degradation (smaller is more robust). Shaded row uses

\mathcal{L}_{\text{CL}}

			SingerID			VocID			InstCls			ESC-50
Upstream	Dist.	Loss	cln	nsy	$\Delta$	cln	nsy	$\Delta$	cln	nsy	$\Delta$	cln	nsy	$\Delta$
DistilHuBERT	–	$\mathcal{L}_{\text{KD}}$	68.04	36.41	31.63	62.19	55.88	6.31	61.42	22.75	38.67	68.24	52.30	15.94
DistilHuBERT	Both	$\mathcal{L}_{\text{KD}}$	57.84	42.21	15.63	59.79	56.68	3.11	60.64	33.45	27.19	67.30	59.40	7.90
DistilHuBERT	Both	$\mathcal{L}_{\text{CL}}$	54.04	41.46	12.58	62.90	58.43	4.47	60.01	37.85	22.16	70.30	64.69	5.61

Table 5: Cross-domain music and environmental-sound results under clean and noisy (CHiME-3, 10 dB SNR) conditions. All metrics Acc%

\uparrow

;

\Delta=

clean

-

noisy. The first row is vanilla DistilHuBERT without noise augmentation; the lower block is the HuBERT teacher under independent noise (Setup 2). Shaded row uses

\mathcal{L}_{\text{CL}}

. GenreID is omitted because the GTZAN benchmark is known to be unreliable.

Model	Noise sampling	LR-768 (%)
DistilHuBERT (no noise aug.)	N/A	96.88
Setup 2 $\mathcal{L}_{\text{CL}}$	Same	95.38
Setup 2 $\mathcal{L}_{\text{KD}}$	Independent	76.98
Setup 2 $\mathcal{L}_{\text{CL}}$	Independent	55.55

Table 6: Same-noise control. LR-768 noise-classification accuracy (%). Chance is 25%.

Model	Noise	LR-768 (%)	Max (%)	Mean (%)	Dims $>$ 30%
DistilHuBERT (no aug.)	N/A	96.88	–	–	727
$\mathcal{L}_{\text{CL}}$	Same	95.4	58.1	39.6	719
$\mathcal{L}_{\text{KD}}$	Indep.	77.0	31.4	26.6	16
$\mathcal{L}_{\text{CL}}$	Indep.	55.5	35.2	26.9	34
CC-only ( $\lambda_{\text{sc}}\!=\!0$ )	Indep.	55.0	30.5	26.3	5

Table 7: Per-dimension noise analysis. Rows are Setup 2 models plus a no-augmentation reference (first row). LR-768 is total noise-classification accuracy. “Dims

>

30%” counts the 768 feature dimensions individually above the 30% threshold.

Figure 1: Per-dimension noise-classification accuracy on the 768 feature dimensions, ranked from most to least noise-informative (chance 25%). At matched total noise near 55%, the full loss $\mathcal{L}_{\text{CL}}$ spreads noise over many more individually informative dimensions than the cross-correlation-only model, while carrying the same total. Identical noise saturates almost every dimension.

	IC (Acc $\uparrow$ )		KS (Acc $\uparrow$ )		ASR (WER $\downarrow$ )		ER (Acc $\uparrow$ )
Loss	cln	nsy	cln	nsy	cln	nsy	cln	nsy
$\mathcal{L}_{\text{KD}}$	93.03	89.29	95.41	93.15	16.98	20.47	60.73	59.03
CC-only ( $\lambda_{\text{sc}}\!=\!0$ )	95.60	90.83	95.16	94.31	16.40	20.83	62.13	60.38
Full $\mathcal{L}_{\text{CL}}$	96.61	94.06	95.59	94.38	15.35	18.18	64.71	63.09

Table 8: Self-correlation ablation (Setup 2, HuBERT teacher). CC-only sets

\lambda_{\text{sc}}\!=\!0

; full

\mathcal{L}_{\text{CL}}

uses

\lambda_{\text{sc}}\!=\!5\times 10^{-6}

. cln/nsy = clean/noisy.

Self-Supervised Knowledge Distillation

Self-Supervised Knowledge Distillation transfers useful model behavior or representations between teacher and student without labeled targets.

See all 22 papers

Correlation-Based Distillation Yields More Mergeable Speech-Music Encoders
Oct 2, 2026Fabian Ritter-GutierrezAudio Representation Learning Knowledge Distillation
Nanyang Technological University, Singapore
HARNESS: Lightweight Distilled Arabic Speech Foundation Models
Mar 31, 2026Vrunda N. Sukhadia, Shammur Absar ChowdhurySpeech Foundation Models Speech Processing
Amazon India · Qatar Computing Research Institute, HBKU, Qatar
Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding
Sep 16, 2026Benedikt Mayrhofer, Enrique Orozco Olivares, Franz Pernkopf +2Audio Representation Learning Streaming ASR
Signal Processing and Speech Communication Laboratory, Graz University of Technology · Comprehensive Center for AI in Medicine, Medical University of Vienna · Dept. of Otolaryngology, Head & Neck Surgery, Div. of Phoniatrics-Logopedics, Speech & Hearing Sci. Lab, Med. Univ. of Vienna

Oct 2, 2026J/K move · Enter open · S save

Correlation-Based Distillation Yields More Mergeable Speech-Music Encoders

Fabian Ritter-Gutierrez

Nanyang Technological University, Singapore

Audio Representation Learning Knowledge Distillation Model Merging

Open paper AskPDF arXiv

A single compact encoder for both speech and music removes the need to maintain a separate model per domain. A practical recipe distils a speech teacher and a music teacher into two small students, then merges them into one model. Two merging algorithms exist: interpolating the student weights from a shared initialisation, or permuting their channels into alignment before averaging. Prior work fixed the distillation loss in both, leaving open whether the loss itself affects how well the students merge. We show that it does. Holding architecture and evaluation fixed, we replace the DistilHuBERT loss \Lkd{} with a correlation-based loss \Lcl{}. Under weight interpolation, \Lcl{} students stay closer to their own better endpoint in

17

18

comparisons and lead on the speech tasks at every interpolated weight. Under activation permutation, they need fewer channels rearranged at three layers, their matched channels correlate more strongly, and they again lead on speech. The two merging algorithms share no mechanism, yet both improve under the same loss substitution. This suggests that the gain lies in the representation \Lcl{} produces rather than in either algorithm.

Abstract

Self-Supervised Knowledge Distillation

Explore similar work

Correlation-Based Distillation Yields More Mergeable Speech-Music Encoders

HARNESS: Lightweight Distilled Arabic Speech Foundation Models

Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding