Self-supervised speech models commonly consist of a set of convolutional neural network (CNN) layers used as a local feature extractor, followed by transformer layers. The CNN layers are computationally demanding, and are also similar across models and resemble the much simpler mel spectral features. We hypothesize that the CNN layers can be replaced by a simple, universal model. To test this hypothesis, we learn a two-layer feature extractor via distillation from one or more teacher models (HuBERT, data2vec, and WavLM). Experiments on SUPERB(-SG) benchmark tasks show that our model largely retains the performance of the teacher models and can also be used in an unseen model (wav2vec 2.0). Our feature extractor requires only 3% of the MACs of the CNNs, resulting in a >20% overall inference runtime reduction.
Figures & tables
Figure 1: Our proposed local feature extractor (model “foot”) replaces the CNN + projection block in speech encoders, and is learned via distillation from one or multiple teacher models.
Figure 2: SUPERB(-SG) test set results (see footnote 2 for task abbreviations). Error rates are converted to accuracies, so higher values are always better. “mel filter banks” uses mel features with an added projection to the transformer, trained with downstream tasks; it is not applied to QbE since there is no QbE training data. “original foot (CNN)” refers to the original model with CNN layers; “single teacher foot” replaces the CNN with a foot distilled from the same model; “multi-teacher foot” replaces the CNN with a foot distilled from all three teachers. “original foot (CNN)” results are reproduced by us and may differ slightly from the publicly reported results. Top: Full raw test results for HuBERT. Bottom: Relative performance improvement when using the original models with their respective foot. Black-bordered markers indicate values beyond the axis range.
Figure 3: PWCCA similarity between teacher model layers. Each curve corresponds to a pair of models. Ci : CNN layer i , P : Projection layer, Tj : Transformer layer j .
PR
ASR (WER ↓ )
KS
QbE
SID
ASV
SD
ER
IC
SF
ST
SE
SS
CLT
SPS
PER ↓
clean
other
ACC ↑
MTWV ↑
ACC ↑
EER ↓
DER ↓
ACC ↑
ACC ↑
F1 ↑
CER ↓
BLEU ↑
PESQ ↑
STOI ↑
SI-SDRi ↑
es WER ↓
zh CER ↓
ar WER ↓
WER ↓
MelHuBERT
8.5
9.7
28.5
88.1
1.7
75.1
6.2
7.2
60.0
84.6
84.9
31.8
11.2
2.9
94.6
10.1
42.3
28.6
57.8
82.0
HuBERT
8.1
9.9
29.7
95.4
7.4
73.1
7.3
8.0
62.0
89.4
82.1
34.1
11.1
2.9
94.6
9.0
43.4
28.5
57.0
74.7
Table 1: Extended evaluation of Stage-2 MelHuBERT-20ms (from scratch) and HuBERT. † † footnotemark:
Figure 4: Results of continued pre-training for HuBERT and wav2vec 2.0 on selected tasks from SUPERB(-SG). Metrics and markers are as in Figure 2 .
Figure 5: Results of pre-training a new wav2vec 2.0 transformer +quantizer, on SUPERB ASR clean dev (no language model).
Large self-supervised speech (SSL) models achieve strong downstream performance, but their size limits deployment in resource-constrained settings. We present HArnESS, an Arabic-centric self-supervised speech model family trained from scratch with iterative self-distillation, together with lightweight student variants that offer strong accuracy-efficiency trade-offs on Automatic Speech Recognition (ASR), Dialect Identification (DID), and Speech Emotion Recognition (SER). Our approach begins with a large bilingual Arabic-English teacher and progressively distills its knowledge into compressed student models while preserving Arabic-relevant acoustic and paralinguistic representations. We further study PCA-based compression of the teacher supervision signal to better match the capacity of shallow and thin students. Compared with HuBERT and XLS-R, HArnESS consistently improves performance on Arabic downstream tasks, while the compressed models remain competitive under substantial structural reduction. These results position HArnESS as a practical and accessible Arabic-centric SSL foundation for real-world speech applications.
Vrunda N. Sukhadia, Shammur Absar Chowdhury
Amazon India · Qatar Computing Research Institute, HBKU, Qatar
General audio foundation models have recently achieved remarkable progress, enabling strong performance across diverse tasks. However, state-of-the-art models remain extremely large, often with hundreds of millions of parameters, leading to high inference costs and limited deployability on edge devices. Knowledge distillation is a proven strategy for model compression, but prior work in audio has mostly focused on supervised settings, relying on class logits, intermediate features, or architecture-specific techniques. Such assumptions exclude models that output only embeddings, such as self-supervised or metric-learning models. We introduce S-SONDO (Self-Supervised KnOwledge DistillatioN for General AuDio FOundation Models), the first framework to distill general audio models using only their output embeddings. By avoiding the need for logits or layer-level alignment, S-SONDO is architecture-agnostic and broadly applicable to embedding-based teachers. We demonstrate its effectiveness by distilling two audio foundation models into three efficient students that are up to 61 times smaller while retaining up to 96% of teacher performance. We also provide practical insights on loss choice and clustering-based balanced data sampling. Code is available here: https://github.com/MedAliAdlouni/ssondo.
Mohammed Ali El Adlouni, Aurian Quelennec, Pierre Chouteau +2
LTCI, T´el´ecom Paris, Institut Polytechnique de Paris, Palaiseau, France
Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to 18.5% of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual speech data to recover performance degradation caused by the zero-shot layer pruning. Mean WER across four languages increases to 20.1% after distillation, compared to 21.9% zero-shot, going from a baseline of 18.2%. We release all of our code (https://github.com/rasgaard/whisper-encoder-layer-prune) and the pruned model (https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).