Pre-trained speech models build entangled internal representations, which capture a variety of features (e.g., pitch, loudness, syntax, or semantics of an utterance) in a distributed encoding. This entanglement makes it difficult to track how such representations rely on textual versus acoustic information when used in downstream applications, limiting their interpretability and transparency. In this paper, we explore an approach to post-hoc disentanglement that builds on the Information Bottleneck principle. We take representations from pre-trained speech encoders as input, and separate them into two distinct components in a cascaded fashion: first we learn a vector encoding content (i.e., what can be transcribed as text), and nothing else; second, we learn a vector encoding all complementary acoustic features relevant to a target task. We apply and evaluate our proposed setup to emotion recognition and speaker identification target tasks, quantifying the relative contribution of textual and acoustic features at each model layer. Finally, we use the approach for disentangled feature attribution as it allows us to identify the most salient speech frames from both the textual and acoustic perspectives.
Figures & tables
Figure 1: We take the original hidden states from a pre-trained speech model and separate them into text and acoustic components. The textual component achieves ASR performance on par with the original hidden states (measured by WER), yet reduces the decodability of acoustic features (e.g., speaker gender) to chance levels. Conversely, the acoustic component exhibits random WER while effectively predicting gender, with its decodability varying depending on the target task (thus satisfying our third desideratum). Scores are averaged over the test set; see Table 2 for exhaustive results.
Figure 2: An overview of our cascaded setup: In stage 1 (text extraction), hidden states derived from a frozen pre-trained speech model are compressed using VIB to decode transcription. In stage 2 (acoustic extraction), the hidden states are compressed to predict the target task, conditioned on the frozen textual latent representations learned in stage 1. The training procedure results in lightweight textual and acoustic encoders that disentangle speech frame representations into textual and task-relevant acoustic features.
Stage 1
Stage 2
Stage 2
Transcription (WER ↓ )
Emotion (Acc. ↑ )
Speaker Id. (Acc. ↑ )
Model
Size
Original
VIB
Original
VIB
Original
VIB
HuBERT
Base
0.45±0.000
0.41±0.006
0.62±0.007
0.63±0.006
0.98±0.001
0.99±0.001
Large
0.35±0.007
0.37±0.041
0.57±0.002
0.66±0.035
0.93±0.003
0.98±0.013
Wav2Vec2
Base
0.50±0.000
0.45±0.002
0.62±0.001
0.59±0.033
0.99±0.000
0.98±0.017
Large
0.47±0.001
0.48±0.034
0.62±0.004
0.66±0.009
0.98±0.004
0.99±0.000
Table 1: Performance of decoders trained on the original hidden states & the VIB latent vectors ( d=128 ). Lower WER and higher Accuracy are better. Random baselines: WER=1 for transcription, Accuracy=0.25 for emotion, and Accuracy=0.04 for speaker identification. Scores are averaged (with ± standard deviation) over three independent runs with different random seeds.
Model
Size
Component
ASR
Mean Intensity
Mean Pitch
Gender
Speaker Identity
(WER ↓ )
(Acc. ↑ )
(Acc. ↑ )
(Acc. ↑ )
(Acc. ↑ )
Random baseline
–
–
1
0.25
0.25
0.50
0.04
HuBERT
Base
Hidden State
0.57
0.53
0.79
1
0.98
ztextual
0.44
0.28
0.27
0.51
0.07
zemotionacoustic
1
0.38
0.42
0.75
0.41
zspeakeracoustic
1
0.56
0.79
1
0.99
Table 2: Probing performances of latent representations ( d=128 ) learned at stages 1 and 2, along with original hidden states derived from different pre-trained models, and components of existing disentanglement frameworks, for transcription (ASR) and a set of audio features (Mean Intensity, Mean Pitch, Gender, and Speaker Identity). For each model and task, components with scores close to the random baseline (i.e., least decodable) are highlighted in bold .
Figure 3: t-SNE of the textual (left) and acoustic (right) latent representations for Wav2Vec2-Large, marked and colored according to their transcription and emotion labels, respectively.
Figure 4: Layerwise performance of VIB ( d=128 ) for transcription at stage 1 ( left ) and emotion classification at stage 2 ( middle ), compared to layerwise performance using original hidden state activations. Right : layerwise textual and acoustic contribution to emotion classification, compared to layerwise performance using hidden states of Base models. Lower WER and higher accuracy are better. The black horizontal dashed line represents the random baseline.
Method
Intensity
Pitch
Sentiment
Textual Attention
0.15
0.07
0.15
Acoustic Attention
0.41
0.15
0.05
Integrated Gradient
0.20
0.07
0.05
Table 3: The alignment (Dot product) of attribution scores and different acoustic and textual cues.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Component
ASR
Mean Intensity
Mean Pitch
Gender
Speaker Identity
(WER ↓ )
(Acc. ↑ )
(Acc. ↑ )
(Acc. ↑ )
(Acc. ↑ )
Random
–
1
0.25
0.25
0.50
0.04
Wav2Vec2
Hidden State
0.61
0.63
0.79
1
1
ztextual
0.48
0.25
0.27
0.55
0.07
zacoustic
1
0.39
0.44
0.77
0.40
ztextual w/o
0.34
0.39
0.65
0.91
0.90
Appendix
Table 4: Probing performances of latent representations learned at stages 1 and 2 with and without information loss constraint (marked by w/o), along with hidden states derived from Wav2Vec2-Base model. For each model and task, components with scores close to the random baseline (i.e., least decodable) are highlighted in bold . Conversely, scores that are expected to be random but exhibit non-random behavior (indicating overlapping information) are underlined .
Transcription (WER ↓ )
Emotion (Acc. ↑ )
Speaker Id. (Acc. ↑ )
Model
Size
Cascaded VIB (Stage 1)
Joint VIB (w/ Emotion)
Joint VIB (w/ Speaker Id.)
Cascaded VIB (Stage 2)
Joint VIB (xxxxx x)
Cascaded VIB (Stage 2)
Joint VIB (xxxxx x)
HuBERT
Base
0.41
0.64
0.92
0.63
0.62
0.99
0.98
Large
0.37
0.35
0.44
0.66
0.59
0.98
0.98
Wav2Vec2
Base
0.45
1
1
0.59
0.56
0.98
0.95
Large
0.48
0.49
0.86
0.66
0.59
0.99
0.99
HuBERT-FT
Large
0.08
0.16
0.33
0.65
0.64
0.98
0.88
Appendix
Table 5: Comparison of cascaded VIB with joint VIB . Random baselines: WER=1 (transcription), Accuracy=0.25 (emotion), Accuracy=0.04 (speaker identification).
Figure 5: The effect of varying constant β values compared to incremental β schedule on training VIB for transcription (stage 1) and emotion (stage 2).
Figure 6: Layer weights of Base (top) and Large (bottom) models, trained for transcription (stage 1), emotion and speaker identity (stage 2), compared with probes applied to the original hidden states.
Stage 2
Speaker Identification (Acc. ↑ )
Model
Original
VIB
HuBERT
0.70
0.87
Wav2Vec2
0.57
0.58
Appendix
Table 6: Performance of linear decoders trained on the original hidden representations and on the VIB latent vectors ( d=128 ) for Speaker Identification on the VoxCeleb dataset. The random baseline accuracy for the 40-way classification task is 0.025 .
Representation disentanglement studies how to separate distinct factors of variation in learned features so models can interpret, control, and transfer them.
Speech content representations are central to voice conversion, speech-to-speech translation, and multimodal language models, yet they are rarely compared under a common generative framework that directly measures what each representation contains. We address this by training a generative model conditioned solely on each representation and evaluating the generated audio along the content, speaker identity, and prosody axes. Across SSL features, supervised tokens, posteriorgrams, and neural audio codecs, we find two distinct regimes: representations that nearly reconstruct the original audio, and representations that effectively disentangle speaker identity. These results show that disentanglement depends not on supervision alone, but on the interaction between the training objective and the representation's information capacity: supervised representations only disentangle speaker identity when their capacity is sufficiently constrained.
Diego Torres, Axel Roebel, Nicolas Obin
STMS Lab IRCAM, CNRS, Sorbonne Universit´e Paris, France
Probing is widely used to study which features can be decoded from language model representations. However, the common decoding probe approach has two limitations that we aim to solve with our new encoding probe approach: contributions of different features to model representations cannot be directly compared, and feature correlations can affect probing results. We present an Encoding Probe that reverses this direction and reconstructs internal representations of models using interpretable features. We evaluate this method on text and speech transformer models, using feature sets spanning acoustics, phonetics, syntax, lexicon, and speaker identity. Our results suggest that speaker-related effects vary strongly across different training objectives and datasets, while syntactic and lexical features contribute independently to reconstruction. These results show that the Encoding Probe provides a complementary perspective on interpreting model representations beyond decodability.
Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception. However, since their representations are inherently black-box, it remains unclear what drives their alignment with brain responses. To remedy this, we built linear encoding models from six interpretable feature families: mel-spectrogram, Gabor filter bank features, speech presence, phonetic, syntactic, and semantic features, and contextualized embeddings from three state-of-the-art SFMs (Whisper, HuBERT, WavLM), quantifying electrocorticography (ECoG) response variance shared between feature classes. Variance-partitioning analyses revealed several key insights: First, the SFMs' alignment with the brain can be mostly explained by their ability to learn and encode simple interpretable speech features. Second, SFMs exhibit a systematic trade-off between encoding of brain-relevant low-level and high-level features across layers. Finally, our results show that SFMs learn brain-relevant semantics which cannot be explained by lower-level speech features, with this capacity increasing with model size and context length. Together, our findings suggest a principled approach to build more interpretable, accurate, and efficient encoding models of the brain by augmenting SFM embeddings with interpretable features.
Riki Shimizu, Richard J. Antonello, Chandan Singh +1
Mortimer B. Zuckerman Mind Brain Behavior Institute Columbia University · Microsoft Research