Speech foundation models trained with self-supervised learning produce generic speech representations that support a wide range of speech processing tasks. When further adapted with supervised learning, these models can achieve strong performance on specific downstream tasks. Recent post-training approaches, such as SAMU-XSLR and SONAR, align speech representations with utterance-level semantic representations, enabling effective multimodal (speech-text) and multilingual applications. While speech foundation models typically learn contextual embeddings at the acoustic frame level, these methods learn representations at the utterance level. In this work, we extend this paradigm to arbitrary utterance-level attributes and propose a unified post-training framework that enables a single speech foundation model to generate multiple types of utterance-level representations. We demonstrate the effectiveness of this approach by jointly learning semantic and speaker representations and evaluating them on multilingual speech retrieval and speaker recognition tasks.
Figures & tables
Figure 1: Attribute-specific branch used to learn an utterance-level representation for attribute τ in the teacher–student framework. Layer representations from the shared SSL encoder are projected, combined using learnable interpolation weights λτ,ℓ , and aggregated by attention pooling to produce an embedding aligned with the teacher representation.
X Speech → EN Speech retrieval
R@1 ↑
fr-en
pl-en
nl-en
es-en
hr-en
de-en
ro-en
cs-en
SONAR
91.91
95.79
95.16
95.30
52.16
94.18
94.45
95.62
Att(sem)
96.55
96.46
95.75
96.48
96.5
94.71
96.83
96.70
Att(sem+spk)
95.94
95.67
95.37
96.01
95.9
93.91
96.49
96.32
EN Speech → Y Speech retrieval
R@1 ↑
en-fr
en-pl
en-nl
en-es
en-hr
en-de
en-ro
en-cs
Table 1: R@1 scores speech → speech translation retrieval for various language pairs (VoxPopuli)
X Speech → EN Text
X Speech → Y Text
R@1 ↑
it-en
fr-en
pt-en
ru-en
it-es
es-it
es-fr
fr-pt
pt-es
SONAR
89.01
82.45
85.05
84.76
92.25
88.57
87.76
84.27
87.08
Att(sem)
90.69
87.01
86.69
83.06
94.35
86.83
87.35
90.08
89.31
Att(sem+spk)
90.10
86.14
85.68
82.27
94.15
86.57
86.91
89.68
88.63
Table 2: R@1 scores speech → text translation retrieval for various language pairs (MTEDx)
X Speech → EN Text
X Speech → Y Text
R@1 ↑
ml-en
lb-en
uz-en
bs-en
my-en
ny-ces
sd-fr
xh-ar
SONAR
22.36
39.98
54.12
27.27
8.19
18.18
21.20
16.07
Att(sem)
62.59
60.88
55.40
60.39
14.11
27.00
59.74
36.67
Att(sem+spk)
61.55
59.59
55.17
60.28
16.38
25.24
58.24
35.71
Table 3: R@1 scores speech → text translation retrieval for various language pairs (FLEURS)
Model
EER ↓
MinDCF0.01↓
ECAPA-TDNN
0.90
0.1104
Att(spk)
0.93
0.1285
Att(sem+spk)
0.91
0.1253
Table 4: Speaker verification results on VoxCeleb1-O
Figure 2: Learned layer-interpolation weights for the semantic and speaker branches of the unified speech encoder.
Recent post-training approaches have shown that self-supervised speech encoders can be adapted to learn a single utterance-level speech attribute through knowledge distillation. In this work, we investigate whether a single speech encoder can simultaneously preserve multiple complementary utterance-level speech representations. We propose a multi-attribute multi-teacher distillation framework that learns semantic, speaker, language, and emotion representations from four frozen teacher models using a shared SSL backbone and attribute-specific branches. Experiments on multilingual semantic retrieval, speaker verification, language identification, and speech emotion recognition show that the proposed model achieves performance close to dedicated single-task students while requiring only a single shared encoder. Furthermore, the learned layer aggregation strategies reveal that different speech attributes naturally emerge at different depths of the SSL encoder, providing new insights into the organization of speech representations.
Speech foundation models, pre-trained on large corpora of unlabelled speech data, produce general-purpose representations which are useful across tasks. However, these representations encode information about salient speech variables in a distributed manner, while downstream speech tasks rely on only some of this variability. In this work, we propose a post-training refinement approach using interventional contrastive learning. By leveraging an interventional dataset and multi-part contrastive loss, we learn a transformation from the entangled representation space of speech foundation models into separate content and speaker subspaces. We evaluate the learnt representations on speaker verification and keyword spotting tasks, showing improved out-of-domain speaker verification performance and evidence that speaker and content information are separated across the learned subspaces.
Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the basis for tokens in speech-aware language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words. However, discriminative ability for words need not imply specialized representation of words per se. Good discrimination of words may be explained by good encoding of word form (phonemes) rather than form-independent word representations encoding identity or syntactic/semantic properties. By partialling out phoneme information using residualization, we show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content. We show that this simple approach to disentanglement can enhance higher-order linguistic information in word discovery tasks.