A single compact encoder for both speech and music removes the need to maintain a separate model per domain. A practical recipe distils a speech teacher and a music teacher into two small students, then merges them into one model. Two merging algorithms exist: interpolating the student weights from a shared initialisation, or permuting their channels into alignment before averaging. Prior work fixed the distillation loss in both, leaving open whether the loss itself affects how well the students merge. We show that it does. Holding architecture and evaluation fixed, we replace the DistilHuBERT loss \Lkd{} with a correlation-based loss \Lcl{}. Under weight interpolation, \Lcl{} students stay closer to their own better endpoint in 17 of 18 comparisons and lead on the speech tasks at every interpolated weight. Under activation permutation, they need fewer channels rearranged at three layers, their matched channels correlate more strongly, and they again lead on speech. The two merging algorithms share no mechanism, yet both improve under the same loss substitution. This suggests that the gain lies in the representation \Lcl{} produces rather than in either algorithm.
Figures & tables
Component
Setting
Distillation data
LibriSpeech 960 h, both students
Teachers
HuBERT Base; MERT ( MERT-v0-public )
Optimiser
Adam, 2×10−4 , 10k linear warm-up
Batch / steps
24 / 200 k
Checkpoint
lowest pre-training loss on dev-clean
LCL weights
λcc=5×10−5 , λsc=5×10−6 , γ=1
Table 1: Configuration shared by every student. Within a pair the students differ only in the teacher and, where stated, the loss. Source: distillation configuration of [ 15 ] ; permutation settings of [ 16 ] .
2-layer
3-layer
Wide
Domain
λ1
LKD
LCL
LKD
LCL
LKD
LCL
Speech
0.9
−0.30
−0.63
−0.78
−0.33
−0.41
−0.40
0.8
−2.33
−1.80
−3.01
−1.01
−1.56
−1.38
0.5
−9.74
−6.89
−9.82
−4.76
−9.70
−9.39
Music
0.9
−1.40
−0.73
−4.94
−0.91
−2.53
−1.13
0.8
−1.76
−0.47
−4.51
−0.59
−2.10
−1.32
Table 2: Mean deviation Δ of ( 7 ) from the better single-domain endpoint under task arithmetic (percentage points; closer to zero is more robust).
Speech
Music & audio
Scores
λ1
Loss
ASR
KS
IC
ER
SingerID
VocID
InstCls
GenreID
ESC-50
Sp.Sc
Mu.Sc
Avg
WER ↓
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Acc ↑
Acc ↑
↑
↑
↑
Ensemble distillation (reference)
–
LKD
15.14
94.19
85.55
60.76
73.07
70.65
64.09
63.22
72.35
901.1
954.2
927.7
Task arithmetic (TA)
1.0
LKD
13.18
96.04
94.70
61.49
69.04
64.30
63.62
63.45
70.65
944.1
920.8
932.5
Table 3: Per-task results for the two-layer students ( D=768 , HuBERT initialisation) under both merging algorithms. The rows λ1=1.0 and 0.0 are the unmerged students, from which the Δ of ( 7 ) is measured; the ensemble baseline is λ -invariant. Scores follow ( 6 ) and bold marks the higher Speech Score in each pair.
2-layer
3-layer
Merge point
LKD
wide LCL
LKD
LKD
LCL
Initialisation
shared
shared
different
shared
shared
CNN 1
41.2
40.2
34.4
42.0
21.3
CNN 2–3
0.0
0.0
99.0 – 99.6
0.0
0.0
CNN 4–5
7.6 – 28.9
0.4 – 2.1
99.4 – 99.6
10.9 – 33.6
0.0
CNN 6–7
70.1 – 85.7
40.8 – 68.2
99.6 – 100
72.1 – 84.0
37.3 – 68.2
Table 4: Percentage of channels CP permutes at each merge point; lower means the students already share more structure. The wide LCL pair also differs in width, head structure and initialisation; the different-initialisation column shows what unaligned students look like.
Figure 1: Difference between the two losses, LCL minus LKD , in Speech Score (left) and Average Score (right); positive bars favour LCL . TA is shown at λ1∈{0.9,0.8,0.5} and CP at {0.9,0.8} , the weights at which CP was run. The wide configuration has no CP bars, since its LKD pair was not merged with CP.
Figure 2: Average Score ( 6 ) for correlation-permutation (CP), task arithmetic (TA) and the ensemble-distillation baseline across five configurations at λ1=0.9 and λ1=0.8 . The two post-hoc methods track each other closely at λ1=0.9 . At λ1=0.8 two configurations separate, the wide pair and the two-layer LKD pair, where TA exceeds CP by 14.0 and 8.8 points (Section 7 ).
Aspect
Ensemble distill.
Task arithmetic
Corr.-perm.
Training
Joint distill.
Two runs
None
Shared init
N/A
Required
Not required
Post-hoc λ
No
Yes
Yes
Model size
Any
Compressed
Any
Calibration
Training data
Not needed
∼ 10K samples
Merge compute
GPU-days
Seconds
Minutes
Table 5: Operating points of the three merging methods.
Self-supervised learning (SSL) speech models are accurate but large. Knowledge distillation compresses them, but the student loses the teacher's noise robustness. Correlation-based distillation addresses this with two terms: cross-correlation aligning student and teacher's representations, and self-correlation decorrelating the student's features. Both the original method and De'HuBERT credited the self-correlation term without isolating it. We show the opposite. On LibriSpeech-100 distillation with held-out CHiME-3 noise at 10,dB, an unbiased full-dimensional probe, a Pearson-variance decomposition, a same-noise causal control, and a per-dimension analysis identify the cross-correlation diagonal as the mechanism that encourages noise invariance, lowering noise-classification accuracy from 76.98% to 55.02%, whereas adding the self-correlation term back leaves it at 55.55%. The self-correlation term instead reorganises the feature space and improves accuracy on the clean downstream tasks, but removes essentially no noise. Across nine speech and music tasks this yields a concrete design rule for distillation: weight the cross-correlation term for noise robustness, tune the self-correlation term for better downstream performance, and sample teacher and student noise independently.
Fabian Ritter-Gutierrez, Nancy F. Chen, Eng Siong Chng
Nanyang Technological University, Singapore · Institute for Infocomm Research (I2R), Singapore
Recent post-training approaches have shown that self-supervised speech encoders can be adapted to learn a single utterance-level speech attribute through knowledge distillation. In this work, we investigate whether a single speech encoder can simultaneously preserve multiple complementary utterance-level speech representations. We propose a multi-attribute multi-teacher distillation framework that learns semantic, speaker, language, and emotion representations from four frozen teacher models using a shared SSL backbone and attribute-specific branches. Experiments on multilingual semantic retrieval, speaker verification, language identification, and speech emotion recognition show that the proposed model achieves performance close to dedicated single-task students while requiring only a single shared encoder. Furthermore, the learned layer aggregation strategies reveal that different speech attributes naturally emerge at different depths of the SSL encoder, providing new insights into the organization of speech representations.
System-wide Dictation on Apple devices runs entirely on-device, and the speech it transcribes reaches the foundation model through a tokenizer: an encoder that maps short windows of waveform onto the representation the language model reads. Because that model is sparsely activated under Instruction-Following Pruning, only a small subset of its experts occupies DRAM at any time, so the always-on tokenizer competes for the same memory, and its parameter count bears directly on power and latency. In this work we study how to compress such a tokenizer by distillation, taking as the supervision target neither the discrete token nor the output distribution but the pre-quantizer latent the model actually consumes - the last representation the two token interfaces share. We train only the student encoder to regress the teacher's per-frame latent under a squared-error objective, with a single affine layer absorbing the teacher-student width mismatch. Because the target precedes both the quantizer and the language-model bridge, one recipe covers both token interfaces we support, and applies both to a tokenizer pretrained alone and to one jointly trained with a language model. At 2.8x compression the distilled student stays within 1.9% relative WER of its teacher on five of six teacher-student pairs without any fine-tuning, and improves on an independently trained tokenizer of identical capacity by 3.9% relative.
Prasanth Yadla, Mohammad Samragh, Dongseong Hwang +7