Correlation-Based Distillation Yields More Mergeable Speech-Music Encoders

Fabian Ritter-Gutierrez

cs.SDOct 2, 2026

Correlation-Based Distillation Yields More Mergeable Speech-Music Encoders

Authors: Fabian Ritter-Gutierrez

Organizations: Nanyang Technological University, Singapore

Abstract

A single compact encoder for both speech and music removes the need to maintain a separate model per domain. A practical recipe distils a speech teacher and a music teacher into two small students, then merges them into one model. Two merging algorithms exist: interpolating the student weights from a shared initialisation, or permuting their channels into alignment before averaging. Prior work fixed the distillation loss in both, leaving open whether the loss itself affects how well the students merge. We show that it does. Holding architecture and evaluation fixed, we replace the DistilHuBERT loss \Lkd{} with a correlation-based loss \Lcl{}. Under weight interpolation, \Lcl{} students stay closer to their own better endpoint in $17$ of $18$ comparisons and lead on the speech tasks at every interpolated weight. Under activation permutation, they need fewer channels rearranged at three layers, their matched channels correlate more strongly, and they again lead on speech. The two merging algorithms share no mechanism, yet both improve under the same loss substitution. This suggests that the gain lies in the representation \Lcl{} produces rather than in either algorithm.

Figures & tables

Component	Setting
Distillation data	LibriSpeech 960 h, both students
Teachers	HuBERT Base; MERT ( MERT-v0-public )
Optimiser	Adam, $2{\times}10^{-4}$ , 10k linear warm-up
Batch / steps	$24$ / $200$ k
Checkpoint	lowest pre-training loss on dev-clean
$\mathcal{L}_{\text{CL}}$ weights	$\lambda_{\text{cc}}{=}5{\times}10^{-5}$ , $\lambda_{\text{sc}}{=}5{\times}10^{-6}$ , $\gamma{=}1$

Table 1: Configuration shared by every student. Within a pair the students differ only in the teacher and, where stated, the loss. Source: distillation configuration of [ 15 ] ; permutation settings of [ 16 ] .

		2-layer		3-layer		Wide
Domain	$\lambda_{1}$	$\mathcal{L}_{\text{KD}}$	$\mathcal{L}_{\text{CL}}$	$\mathcal{L}_{\text{KD}}$	$\mathcal{L}_{\text{CL}}$	$\mathcal{L}_{\text{KD}}$	$\mathcal{L}_{\text{CL}}$
Speech	0.9	$\mathbf{-0.30}$	$-0.63$	$-0.78$	$-0.33$	$-0.41$	$-0.40$
	0.8	$-2.33$	$-1.80$	$-3.01$	$\mathbf{-1.01}$	$-1.56$	$-1.38$
	0.5	$-9.74$	$-6.89$	$-9.82$	$\mathbf{-4.76}$	$-9.70$	$-9.39$
Music	0.9	$-1.40$	$\mathbf{-0.73}$	$-4.94$	$-0.91$	$-2.53$	$-1.13$
	0.8	$-1.76$	$\mathbf{-0.47}$	$-4.51$	$-0.59$	$-2.10$	$-1.32$

Table 2: Mean deviation

\Delta

of ( 7 ) from the better single-domain endpoint under task arithmetic (percentage points; closer to zero is more robust).

Ensemble distillation (reference)
		Speech				Music & audio					Scores
$\lambda_{1}$	Loss	ASR	KS	IC	ER	SingerID	VocID	InstCls	GenreID	ESC-50	Sp.Sc	Mu.Sc	Avg
		WER $\downarrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	$\uparrow$	$\uparrow$	$\uparrow$
–	$\mathcal{L}_{\text{KD}}$	$15.14$	$94.19$	$85.55$	$60.76$	$73.07$	$70.65$	$64.09$	$63.22$	$72.35$	$901.1$	$954.2$	$927.7$
Task arithmetic (TA)
$1.0$	$\mathcal{L}_{\text{KD}}$	$13.18$	$96.04$	$94.70$	$61.49$	$69.04$	$64.30$	$63.62$	$63.45$	$70.65$	$944.1$	$920.8$	$932.5$

Table 3: Per-task results for the two-layer students (

D{=}768

, HuBERT initialisation) under both merging algorithms. The rows

\lambda_{1}{=}1.0

and

0.0

are the unmerged students, from which the

\Delta

of ( 7 ) is measured; the ensemble baseline is

\lambda

-invariant. Scores follow ( 6 ) and bold marks the higher Speech Score in each pair.

	2-layer			3-layer
Merge point	$\mathcal{L}_{\text{KD}}$	wide $\mathcal{L}_{\text{CL}}$	$\mathcal{L}_{\text{KD}}$	$\mathcal{L}_{\text{KD}}$	$\mathcal{L}_{\text{CL}}$
Initialisation	shared	shared	different	shared	shared
CNN 1	$41.2$	$40.2$	$34.4$	$42.0$	$\mathbf{21.3}$
CNN 2–3	$0.0$	$0.0$	$99.0$ – $99.6$	$0.0$	$0.0$
CNN 4–5	$7.6$ – $28.9$	$0.4$ – $2.1$	$99.4$ – $99.6$	$10.9$ – $33.6$	$\mathbf{0.0}$
CNN 6–7	$70.1$ – $85.7$	$40.8$ – $68.2$	$99.6$ – $100$	$72.1$ – $84.0$	$\mathbf{37.3}$ – $\mathbf{68.2}$

Table 4: Percentage of channels CP permutes at each merge point; lower means the students already share more structure. The wide

\mathcal{L}_{\text{CL}}

pair also differs in width, head structure and initialisation; the different-initialisation column shows what unaligned students look like.

Figure 1: Difference between the two losses,

\mathcal{L}_{\text{CL}}

minus

\mathcal{L}_{\text{KD}}

, in Speech Score (left) and Average Score (right); positive bars favour

\mathcal{L}_{\text{CL}}

. TA is shown at

\lambda_{1}\in\{0.9,0.8,0.5\}

and CP at

\{0.9,0.8\}

, the weights at which CP was run. The wide configuration has no CP bars, since its

\mathcal{L}_{\text{KD}}

pair was not merged with CP.

Figure 2: Average Score ( 6 ) for correlation-permutation (CP), task arithmetic (TA) and the ensemble-distillation baseline across five configurations at

\lambda_{1}{=}0.9

and

\lambda_{1}{=}0.8

. The two post-hoc methods track each other closely at

\lambda_{1}{=}0.9

. At

\lambda_{1}{=}0.8

two configurations separate, the wide pair and the two-layer

\mathcal{L}_{\text{KD}}

pair, where TA exceeds CP by

14.0

and

8.8

points (Section 7 ).

Aspect	Ensemble distill.	Task arithmetic	Corr.-perm.
Training	Joint distill.	Two runs	None
Shared init	N/A	Required	Not required
Post-hoc $\lambda$	No	Yes	Yes
Model size	Any	Compressed	Any
Calibration	Training data	Not needed	$\sim$ 10K samples
Merge compute	GPU-days	Seconds	Minutes

Table 5: Operating points of the three merging methods.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 211 papers

What Actually Makes Correlation-Based SSL Distillation Noise-Robust? A Mechanistic Correction
Oct 2, 2026Fabian Ritter-Gutierrez, Nancy F. Chen, Eng Siong ChngSelf-Supervised Knowledge Distillation Knowledge Distillation
Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), Singapore
Learning Speech Attribute Representations via Multi-Teacher Distillation
Oct 15, 2026Maryem Bouziane, Salima Mdhaffar, Yannick EstèveRepresentation Learning Multi-Teacher Knowledge Distillation
LIA, Avignon Université, Avignon, France
Compressing Streaming Neural Audio Encoders via Latent-Space Distillation
Sep 3, 2026Prasanth Yadla, Mohammad Samragh, Dongseong Hwang +7Audio Token Compression Audio Representation Learning
Apple

Oct 2, 2026J/K move · Enter open · S save

What Actually Makes Correlation-Based SSL Distillation Noise-Robust? A Mechanistic Correction

Fabian Ritter-Gutierrez, Nancy F. Chen, Eng Siong Chng

Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), Singapore

Self-Supervised Knowledge Distillation Knowledge Distillation Audio Representation Learning Self-Supervised Speech Representation Learning

Open paper AskPDF arXiv

Self-supervised learning (SSL) speech models are accurate but large. Knowledge distillation compresses them, but the student loses the teacher's noise robustness. Correlation-based distillation addresses this with two terms: cross-correlation aligning student and teacher's representations, and self-correlation decorrelating the student's features. Both the original method and De'HuBERT credited the self-correlation term without isolating it. We show the opposite. On LibriSpeech-100 distillation with held-out CHiME-3 noise at 10,dB, an unbiased full-dimensional probe, a Pearson-variance decomposition, a same-noise causal control, and a per-dimension analysis identify the cross-correlation diagonal as the mechanism that encourages noise invariance, lowering noise-classification accuracy from 76.98% to 55.02%, whereas adding the self-correlation term back leaves it at 55.55%. The self-correlation term instead reorganises the feature space and improves accuracy on the clean downstream tasks, but removes essentially no noise. Across nine speech and music tasks this yields a concrete design rule for distillation: weight the cross-correlation term for noise robustness, tune the self-correlation term for better downstream performance, and sample teacher and student noise independently.

Abstract

Audio Representation Learning

Explore similar work

What Actually Makes Correlation-Based SSL Distillation Noise-Robust? A Mechanistic Correction

Learning Speech Attribute Representations via Multi-Teacher Distillation

Compressing Streaming Neural Audio Encoders via Latent-Space Distillation