Post-hoc Disentanglement of Textual and Acoustic Features in Pre-trained Speech Encoders

Hosein Mohebbi; Grzegorz Chrupała; Willem Zuidema; Afra Alishahi; Ivan Titov

cs.CLOct 3, 2024

Post-hoc Disentanglement of Textual and Acoustic Features in Pre-trained Speech Encoders

Authors: Hosein Mohebbi, Grzegorz Chrupała, Willem Zuidema, Afra Alishahi, Ivan Titov

Organizations: CSAI, Tilburg University · ILLC, University of Amsterdam · ILCC, University of Edinburgh

Abstract

Pre-trained speech models build entangled internal representations, which capture a variety of features (e.g., pitch, loudness, syntax, or semantics of an utterance) in a distributed encoding. This entanglement makes it difficult to track how such representations rely on textual versus acoustic information when used in downstream applications, limiting their interpretability and transparency. In this paper, we explore an approach to post-hoc disentanglement that builds on the Information Bottleneck principle. We take representations from pre-trained speech encoders as input, and separate them into two distinct components in a cascaded fashion: first we learn a vector encoding content (i.e., what can be transcribed as text), and nothing else; second, we learn a vector encoding all complementary acoustic features relevant to a target task. We apply and evaluate our proposed setup to emotion recognition and speaker identification target tasks, quantifying the relative contribution of textual and acoustic features at each model layer. Finally, we use the approach for disentangled feature attribution as it allows us to identify the most salient speech frames from both the textual and acoustic perspectives.

Figures & tables

Figure 1: We take the original hidden states from a pre-trained speech model and separate them into text and acoustic components. The textual component achieves ASR performance on par with the original hidden states (measured by WER), yet reduces the decodability of acoustic features (e.g., speaker gender) to chance levels. Conversely, the acoustic component exhibits random WER while effectively predicting gender, with its decodability varying depending on the target task (thus satisfying our third desideratum). Scores are averaged over the test set; see Table 2 for exhaustive results.

Figure 2: An overview of our cascaded setup: In stage 1 (text extraction), hidden states derived from a frozen pre-trained speech model are compressed using VIB to decode transcription. In stage 2 (acoustic extraction), the hidden states are compressed to predict the target task, conditioned on the frozen textual latent representations learned in stage 1. The training procedure results in lightweight textual and acoustic encoders that disentangle speech frame representations into textual and task-relevant acoustic features.

		Stage 1		Stage 2		Stage 2
		Transcription (WER $\downarrow$ )		Emotion (Acc. $\uparrow$ )		Speaker Id. (Acc. $\uparrow$ )
Model	Size	Original	VIB	Original	VIB	Original	VIB
HuBERT	Base	$\text{0.45}_{\pm\text{0.000}}$	$\text{0.41}_{\pm\text{0.006}}$	$\text{0.62}_{\pm\text{0.007}}$	$\text{0.63}_{\pm\text{0.006}}$	$\text{0.98}_{\pm\text{0.001}}$	$\text{0.99}_{\pm\text{0.001}}$
HuBERT	Large	$\text{0.35}_{\pm\text{0.007}}$	$\text{0.37}_{\pm\text{0.041}}$	$\text{0.57}_{\pm\text{0.002}}$	$\text{0.66}_{\pm\text{0.035}}$	$\text{0.93}_{\pm\text{0.003}}$	$\text{0.98}_{\pm\text{0.013}}$
Wav2Vec2	Base	$\text{0.50}_{\pm\text{0.000}}$	$\text{0.45}_{\pm\text{0.002}}$	$\text{0.62}_{\pm\text{0.001}}$	$\text{0.59}_{\pm\text{0.033}}$	$\text{0.99}_{\pm\text{0.000}}$	$\text{0.98}_{\pm\text{0.017}}$
Wav2Vec2	Large	$\text{0.47}_{\pm\text{0.001}}$	$\text{0.48}_{\pm\text{0.034}}$	$\text{0.62}_{\pm\text{0.004}}$	$\text{0.66}_{\pm\text{0.009}}$	$\text{0.98}_{\pm\text{0.004}}$	$\text{0.99}_{\pm\text{0.000}}$

Table 1: Performance of decoders trained on the original hidden states & the VIB latent vectors (

d\!=\!128

). Lower WER and higher Accuracy are better. Random baselines:

\text{WER}\!=\!1

for transcription,

\text{Accuracy}\!=\!0.25

for emotion, and

\text{Accuracy}\!=\!0.04

for speaker identification. Scores are averaged (with

\pm

standard deviation) over three independent runs with different random seeds.

Model	Size	Component	ASR	Mean Intensity	Mean Pitch	Gender	Speaker Identity
			(WER $\downarrow$ )	(Acc. $\uparrow$ )	(Acc. $\uparrow$ )	(Acc. $\uparrow$ )	(Acc. $\uparrow$ )
Random baseline	–	–	1	0.25	0.25	0.50	0.04
HuBERT	Base	Hidden State	0.57	0.53	0.79	1	0.98
		$\bm{z}^{\text{textual}}$	0.44	0.28	0.27	0.51	0.07
		$\bm{z}^{\text{acoustic}}_{\text{emotion}}$	1	0.38	0.42	0.75	0.41
		$\bm{z}^{\text{acoustic}}_{\text{speaker}}$	1	0.56	0.79	1	0.99

Table 2: Probing performances of latent representations (

d\!=\!128

) learned at stages 1 and 2, along with original hidden states derived from different pre-trained models, and components of existing disentanglement frameworks, for transcription (ASR) and a set of audio features (Mean Intensity, Mean Pitch, Gender, and Speaker Identity). For each model and task, components with scores close to the random baseline (i.e., least decodable) are highlighted in bold .

Figure 3: t-SNE of the textual (left) and acoustic (right) latent representations for Wav2Vec2-Large, marked and colored according to their transcription and emotion labels, respectively.

Figure 4: Layerwise performance of VIB (

d\!=\!128

) for transcription at stage 1 ( left ) and emotion classification at stage 2 ( middle ), compared to layerwise performance using original hidden state activations. Right : layerwise textual and acoustic contribution to emotion classification, compared to layerwise performance using hidden states of Base models. Lower WER and higher accuracy are better. The black horizontal dashed line represents the random baseline.

Method	Intensity	Pitch	Sentiment
Textual Attention	0.15	0.07	0.15
Acoustic Attention	0.41	0.15	0.05
Integrated Gradient	0.20	0.07	0.05

Table 3: The alignment (Dot product) of attribution scores and different acoustic and textual cues.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Component	ASR	Mean Intensity	Mean Pitch	Gender	Speaker Identity
		(WER $\downarrow$ )	(Acc. $\uparrow$ )	(Acc. $\uparrow$ )	(Acc. $\uparrow$ )	(Acc. $\uparrow$ )
Random	–	1	0.25	0.25	0.50	0.04
Wav2Vec2	Hidden State	0.61	0.63	0.79	1	1
	$\bm{z}^{\text{textual}}$	0.48	0.25	0.27	0.55	0.07
	$\bm{z}^{\text{acoustic}}$	1	0.39	0.44	0.77	0.40
	$\bm{z}^{\text{textual w/o}}$	0.34	0.39	0.65	0.91	0.90

Appendix

Table 4: Probing performances of latent representations learned at stages 1 and 2 with and without information loss constraint (marked by w/o), along with hidden states derived from Wav2Vec2-Base model. For each model and task, components with scores close to the random baseline (i.e., least decodable) are highlighted in bold . Conversely, scores that are expected to be random but exhibit non-random behavior (indicating overlapping information) are underlined .

		Transcription (WER $\downarrow$ )			Emotion (Acc. $\uparrow$ )		Speaker Id. (Acc. $\uparrow$ )
Model	Size	Cascaded VIB (Stage 1)	Joint VIB (w/ Emotion)	Joint VIB (w/ Speaker Id.)	Cascaded VIB (Stage 2)	Joint VIB (xxxxx x)	Cascaded VIB (Stage 2)	Joint VIB (xxxxx x)
HuBERT	Base	0.41	0.64	0.92	0.63	0.62	0.99	0.98
HuBERT	Large	0.37	0.35	0.44	0.66	0.59	0.98	0.98
Wav2Vec2	Base	0.45	1	1	0.59	0.56	0.98	0.95
Wav2Vec2	Large	0.48	0.49	0.86	0.66	0.59	0.99	0.99
HuBERT-FT	Large	0.08	0.16	0.33	0.65	0.64	0.98	0.88

Appendix

Table 5: Comparison of cascaded VIB with joint VIB . Random baselines:

\text{WER}\!=\!1

(transcription),

\text{Accuracy}\!=\!0.25

(emotion),

\text{Accuracy}\!=\!0.04

(speaker identification).

Figure 5: The effect of varying constant

\beta

values compared to incremental

\beta

schedule on training VIB for transcription (stage 1) and emotion (stage 2).

Figure 6: Layer weights of Base (top) and Large (bottom) models, trained for transcription (stage 1), emotion and speaker identity (stage 2), compared with probes applied to the original hidden states.

	Stage 2
	Speaker Identification (Acc. $\uparrow$ )
Model	Original	VIB
HuBERT	0.70	0.87
Wav2Vec2	0.57	0.58

Appendix

Table 6: Performance of linear decoders trained on the original hidden representations and on the VIB latent vectors (

d=128

) for Speaker Identification on the VoxCeleb dataset. The random baseline accuracy for the 40-way classification task is

0.025

Representation Disentanglement

Representation disentanglement studies how to separate distinct factors of variation in learned features so models can interpret, control, and transfer them.

See all 98 papers

A Comprehensive Study of Content Representations for Speech Synthesis
Sep 25, 2026Diego Torres, Axel Roebel, Nicolas ObinAudio Representation Learning Discrete Speech Representation Learning
STMS Lab IRCAM, CNRS, Sorbonne Universit´e Paris, France
Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe
May 1, 2026Gaofei Shen, Martijn Bentum, Tomas O. Lentz +2Transformer Interpretability Representation Probing
Tilburg University · Radboud University
Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models
Jul 21, 2025Riki Shimizu, Richard J. Antonello, Chandan Singh +1Speech Foundation Models Neural Encoding
Mortimer B. Zuckerman Mind Brain Behavior Institute Columbia University · Microsoft Research

Sep 25, 2026J/K move · Enter open · S save

A Comprehensive Study of Content Representations for Speech Synthesis

Diego Torres, Axel Roebel, Nicolas Obin

STMS Lab IRCAM, CNRS, Sorbonne Universit´e Paris, France

Audio Representation Learning Discrete Speech Representation Learning Speaker Disentanglement Speech Generation Evaluation

Open paper AskPDF arXiv

Speech content representations are central to voice conversion, speech-to-speech translation, and multimodal language models, yet they are rarely compared under a common generative framework that directly measures what each representation contains. We address this by training a generative model conditioned solely on each representation and evaluating the generated audio along the content, speaker identity, and prosody axes. Across SSL features, supervised tokens, posteriorgrams, and neural audio codecs, we find two distinct regimes: representations that nearly reconstruct the original audio, and representations that effectively disentangle speaker identity. These results show that disentanglement depends not on supervision alone, but on the interaction between the training objective and the representation's information capacity: supervised representations only disentangle speaker identity when their capacity is sufficiently constrained.

Abstract

Representation Disentanglement

Explore similar work

A Comprehensive Study of Content Representations for Speech Synthesis

Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe

Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models