Learning Speech Attribute Representations via Multi-Teacher Distillation

Maryem Bouziane; Salima Mdhaffar; Yannick Estève

cs.CLOct 15, 2026

Learning Speech Attribute Representations via Multi-Teacher Distillation

Authors: Maryem Bouziane, Salima Mdhaffar, Yannick Estève

Organizations: LIA, Avignon Université, Avignon, France

Abstract

Recent post-training approaches have shown that self-supervised speech encoders can be adapted to learn a single utterance-level speech attribute through knowledge distillation. In this work, we investigate whether a single speech encoder can simultaneously preserve multiple complementary utterance-level speech representations. We propose a multi-attribute multi-teacher distillation framework that learns semantic, speaker, language, and emotion representations from four frozen teacher models using a shared SSL backbone and attribute-specific branches. Experiments on multilingual semantic retrieval, speaker verification, language identification, and speech emotion recognition show that the proposed model achieves performance close to dedicated single-task students while requiring only a single shared encoder. Furthermore, the learned layer aggregation strategies reveal that different speech attributes naturally emerge at different depths of the SSL encoder, providing new insights into the organization of speech representations.

Figures & tables

Fig. 1: Overview of the multi-teacher distillation architecture.

Dataset	Language	# Classes	Hours
ESD [ 20 ]	Mixed	5	23.23
ASVP-ESD [ 21 ]	Mixed	12	11.35
EmoV-DB [ 22 ]	English	5	7.53
MELD [ 23 ]	English	7	6.36
PAVOQUE [ 24 ]	German	5	6.28
SUBESCO [ 25 ]	Bangla	7	6.26

TABLE I: Emotion datasets used for emotion supervision.

Model	X $\rightarrow$ EN							EN $\rightarrow$ Y							X $\rightarrow$ Y
	fr-en	pl-en	nl-en	es-en	hr-en	de-en	ro-en	en-fr	en-pl	en-nl	en-es	en-hr	en-de	en-ro	fr-de	hr-cs	ro-fr	hu-da	de-fr	cs-hr	fr-ro	da-hu
SENSE	96.55	96.46	95.75	96.48	96.50	94.71	96.83	96.54	96.25	95.71	96.37	96.31	94.12	97.16	95.20	94.75	96.55	92.69	95.36	94.69	96.90	92.63
SEM-only	94.05	93.83	92.62	94.62	95.17	91.21	95.90	93.96	94.66	92.77	94.12	95.96	90.65	96.24	90.79	94.03	95.58	89.71	91.94	93.81	96.06	90.73
Multi-Attribute	95.19	94.85	94.55	95.45	95.41	92.68	96.10	95.10	94.85	94.62	95.30	95.52	92.29	96.37	92.43	93.63	95.75	88.41	92.31	93.63	96.07	88.46

TABLE II: Recall@1 scores (%) for speech-to-speech retrieval on VoxPopuli.

Model	ml-en	lb-en	uz-en	bs-en	my-en	ny-ces	sd-fr	xh-ar
SENSE	62.59	60.88	55.40	60.39	14.11	27.00	59.74	36.67
SEM-only	63.11	60.99	56.10	59.52	19.11	29.04	58.67	37.62
Multi-Attribute	62.90	60.66	56.10	59.85	20.59	25.37	57.92	36.19

TABLE III: Recall@1 scores for multilingual retrieval on FLEURS.

Model	EER (%) $\downarrow$	minDCF $\downarrow$
ECAPA-spk	0.899	0.001104
SPK-only	0.899	0.001207
Multi-Attribute	0.915	0.001229

TABLE IV: Speaker verification results on VoxCeleb1-O.

Model	Classification error (%) $\downarrow$
ECAPA-lang teacher	6.58
LANG-only	6.83
Multi-Attribute Lang	7.02
Multi-Attribute Sem + Lang	6.39

TABLE V: Language identification results on VoxLingua107.

Fig. 2: Unweighted Accuracy on speech emotion recognition datasets. The figure compares the emotion2vec-base teacher, the EMO-only student, and the emotion branch of the proposed Multi-Attribute model.

Fig. 3: Layer-interpolation weights learned by the four branches of the Multi-Attribute model. Semantic information is mainly extracted from intermediate layers, while speaker and language information are distributed across several encoder layers.

Panel of Figure 9 — Fig. 4: Layer-interpolation weights learned by the single-task students. Compared with the Multi-Attribute model, these distributions show that joint training modifies the layer usage of each attribute branch.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 798 papers

Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encoder
Mar 9, 2026Maryem Bouziane, Salima Mdhaffar, Yannick EstèveSpeech Processing Audio Representation Learning
LIA - Avignon Université, France
Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding
Sep 16, 2026Benedikt Mayrhofer, Enrique Orozco Olivares, Franz Pernkopf +2Audio Representation Learning Streaming ASR
Signal Processing and Speech Communication Laboratory, Graz University of Technology · Comprehensive Center for AI in Medicine, Medical University of Vienna · Dept. of Otolaryngology, Head & Neck Surgery, Div. of Phoniatrics-Logopedics, Speech & Hearing Sci. Lab, Med. Univ. of Vienna
AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition
Jul 28, 2026Yuqi Li, Yi-Cheng Lin, Xianglong Wang +5Relational Knowledge Distillation Speech Emotion Recognition
Department of Electrical Engineering, The City College of New York · National Taiwan University · Wyze Inc. +1

Mar 9, 2026J/K move · Enter open · S save

Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encoder

Maryem Bouziane, Salima Mdhaffar, Yannick Estève

LIA - Avignon Université, France

Speech Processing Audio Representation Learning Speech Foundation Models

Open paper AskPDF arXiv

Speech foundation models trained with self-supervised learning produce generic speech representations that support a wide range of speech processing tasks. When further adapted with supervised learning, these models can achieve strong performance on specific downstream tasks. Recent post-training approaches, such as SAMU-XSLR and SONAR, align speech representations with utterance-level semantic representations, enabling effective multimodal (speech-text) and multilingual applications. While speech foundation models typically learn contextual embeddings at the acoustic frame level, these methods learn representations at the utterance level. In this work, we extend this paradigm to arbitrary utterance-level attributes and propose a unified post-training framework that enables a single speech foundation model to generate multiple types of utterance-level representations. We demonstrate the effectiveness of this approach by jointly learning semantic and speaker representations and evaluating them on multilingual speech retrieval and speaker recognition tasks.

Abstract

Representation Learning

Explore similar work

Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encoder

Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding

AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition