HARNESS: Lightweight Distilled Arabic Speech Foundation Models

Vrunda N. Sukhadia; Shammur Absar Chowdhury

eess.ASMar 31, 2026

HARNESS: Lightweight Distilled Arabic Speech Foundation Models

Authors: Vrunda N. Sukhadia, Shammur Absar Chowdhury

Organizations: Amazon India · Qatar Computing Research Institute, HBKU, Qatar

Abstract

Large self-supervised speech (SSL) models achieve strong downstream performance, but their size limits deployment in resource-constrained settings. We present HArnESS, an Arabic-centric self-supervised speech model family trained from scratch with iterative self-distillation, together with lightweight student variants that offer strong accuracy-efficiency trade-offs on Automatic Speech Recognition (ASR), Dialect Identification (DID), and Speech Emotion Recognition (SER). Our approach begins with a large bilingual Arabic-English teacher and progressively distills its knowledge into compressed student models while preserving Arabic-relevant acoustic and paralinguistic representations. We further study PCA-based compression of the teacher supervision signal to better match the capacity of shallow and thin students. Compared with HuBERT and XLS-R, HArnESS consistently improves performance on Arabic downstream tasks, while the compressed models remain competitive under substantial structural reduction. These results position HArnESS as a practical and accessible Arabic-centric SSL foundation for real-world speech applications.

Figures & tables

Figure 1: Overview of the iterative self-distillation framework used to build the HArnESS model family.

Category	Sub-category / Dialect	Duration (Hrs)
Original Clean Data		7,566.00
	English Subset	3,565.00
	Arabic Subset	4,001.00
	MSA / General Arabic	3,603.28
	Levantine	107.69
	Egyptian	109.20

Table 1: Comprehensive breakdown of the 23,000-hour training corpus, including language distribution, dialectal variety, and augmentation strategies.

Total pre-training audio (Iter 1–2)	23k hours (Arabic/English $\approx$ balanced)
$K$ -means training subset (Iter 1–2)	300 hours
Distillation audio (Iter 3)	1,100 hours (Arabic-only)
$K$ -means training subset (Iter 3)	30% $\approx$ 300 hours
# clusters ( $K$ )	1000
Feature type for $i{=}0$ targets	MFCC (39-dim)
Embedding layer for $i{=}1$ targets	Layer 9 embeddings of $\mathbf{M}_{0}$

Table 2: Key pre-training hyperparameters.

CNN Encoder
Models	XR	HuL	H-L	H-S	H-ST	H-ST (PCA)
Supervision	–	–	$L9_{\text{emb}}$ ( $i=1$ )	$L23_{\text{emb}}$ ( $i=2$ )		PCA( $L23_{\text{emb}}$ ) ( $i=2$ )
Strides	5, 2, 2, 2, 2, 2, 2
Kernel Width	10, 3, 3, 3, 3, 2, 2
Channels	512
Transformer

Table 3: SSL Model Architecture Comparison. XR: XLS-R, HuL: HuBERT-Large, H-L: HArnESS-Large, H-S: HArnESS-Shallow, H-ST: HArnESS-Shallow and Thin. Dim. dimension, Emb.: Embedding.

L*_{emb}

: Embedding from layer * (e.g. 23) of model from iteration

i

Panel of Figure 2 — Figure 2: Ablation results for the compressed student models. 2(a) ) Effect of student initialization strategy. ( 2(b) ) Effect of reducing the number of attention heads. ( 2(c) ) Effect of applying PCA to teacher embeddings before clustering when generating pseudo-labels for student training. H-ST( $emb_{d}$ , $attn_{h}$ ), SPA means PCA applied for supervision.

Published task-specific reference results (context only)
Models	ASR (WER $\downarrow$ )		SER (Acc $\uparrow$ )	DID (Acc $\uparrow$ )
Models	MGB2	MGB3	KSUEmotion	ADI5
Reference result	10.24 Fanar et al. (2025)	21.31 Fanar et al. (2025)	85.53% Abouzeid et al. (2025)	82.5% Kulkarni and Aldarmaki (2023)
Our downstream evaluation with frozen SSL encoders
HuBERT-L ( English )	22.6 ∗	51.2 ∗	91.92%	64.14%
XLS-R ( Multilingual )	22.60 ∗	51.80 ∗	73.32%	42.35%

Table 4: Performance comparison on ASR, SER, and DID. ASR downstream results in our setup are obtained by training on a 300h QASR subset, results denoted by a ∗ . The top block lists previously published task-specific reference results from separate studies and is included only for context. These numbers are not from a unified baseline and are not directly comparable to the models evaluated in our setup. L denotes the large teacher model, S the shallow student, and ST the shallow-thin student.

\Delta S

denotes overall structural compression relative to HArnESS-L.

Test Sets	$emb_{d}$ =1024	$emb_{d}$ =512	$emb_{d}$ =256
MGB2 (WER ↓)	20.2	23.20	22.3
KSUEmotion (Acc ↑)	91.15%	89.02%	79.42%
ADI5 (Acc ↑)	70.84%	69.77%	53.41%
$\Delta S$	70.43%	91.14%	96.52%

Table 5: Performance Comparison for different embedding dimensions.

\Delta S

: Overall structural compression.

Speech Foundation Models

Speech foundation models are large pretrained systems that learn reusable audio representations for speech understanding, recognition, and generation.

See all 63 papers

Ara-Best-RQ: Multi Dialectal Arabic SSL
Mar 23, 2026Haroun Elleuch, Ryan Whetten, Salima Mdhaffar +2Arabic Dialects Speech Processing
ELYADATA, Paris, France · Laboratoire Informatique d’Avignon, Avignon Université, Avignon, France
End-to-end Jordanian dialect speech-to-text self-supervised learning framework
Sep 21, 2026Ali A. Safieh, Ibrahim Abu Alhaol, Rawan GhnematAutomatic Speech Recognition Self-Supervised Speech Representation Learning
Data Science Department, King Hussein School of Computing Sciences, Princess Sumaya University for Technology, Amman, Jordan
Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding
Sep 16, 2026Benedikt Mayrhofer, Enrique Orozco Olivares, Franz Pernkopf +2Audio Representation Learning Streaming ASR
Signal Processing and Speech Communication Laboratory, Graz University of Technology · Comprehensive Center for AI in Medicine, Medical University of Vienna · Dept. of Otolaryngology, Head & Neck Surgery, Div. of Phoniatrics-Logopedics, Speech & Hearing Sci. Lab, Med. Univ. of Vienna

Mar 23, 2026J/K move · Enter open · S save

Ara-Best-RQ: Multi Dialectal Arabic SSL

Haroun Elleuch, Ryan Whetten, Salima Mdhaffar, Yannick Estève+1

ELYADATA, Paris, France · Laboratoire Informatique d’Avignon, Avignon Université, Avignon, France

Arabic Dialects Speech Processing Self-Supervised Speech Representation Learning Speech Foundation Models Automatic Speech Recognition

Open paper AskPDF arXiv

We present Ara-BEST-RQ, a family of self-supervised learning (SSL) models specifically designed for multi-dialectal Arabic speech processing. Leveraging 5,640 hours of crawled Creative Commons speech and combining it with publicly available datasets, we pre-train conformer-based BEST-RQ models up to 600M parameters. Our models are evaluated on dialect identification (DID) and automatic speech recognition (ASR) tasks, achieving state-of-the-art performance on the former while using fewer parameters than competing models. We demonstrate that family-targeted pre-training on Arabic dialects significantly improves downstream performance compared to multilingual or monolingual models trained on non-Arabic data. All models, code, and pre-processed datasets will be publicly released to support reproducibility and further research in Arabic speech technologies.

Abstract

Speech Foundation Models

Explore similar work

Ara-Best-RQ: Multi Dialectal Arabic SSL

End-to-end Jordanian dialect speech-to-text self-supervised learning framework

Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding