Disentangling Linguistic and Paralinguistic Information with Routed Sparse Autoencoders

Beimnet Bekele Guta; Xiaoyu Yang; Guangzhi Sun; Philip C. Woodland

cs.CLOct 7, 2026

Disentangling Linguistic and Paralinguistic Information with Routed Sparse Autoencoders

Authors: Beimnet Bekele Guta, Xiaoyu Yang, Guangzhi Sun, Philip C. Woodland

Organizations: Department of Engineering, University of Cambridge, Cambridge, UK

Abstract

Self-supervised speech encoders contain linguistic and paralinguistic information in a shared, entangled representation space. We combine a TopK sparse autoencoder with route-specific supervision and cross-factor adversaries. Across frozen SPEAR and WavLM encoders, independent probes show factor-specific retention and suppression: linguistic information remains stronger in the linguistic route, while paralinguistic factors, including speaker identity, emotion, and prosody, are retained in the paralinguistic route and substantially reduced in the linguistic route. The route organisation learned on LibriSpeech persists on MSP-Podcast without representation-side retraining. Feature-space route interventions further transfer the swapped factor while largely preserving the information carried by the unchanged route. These results show consistent route-selective separation across encoders, corpora, independent probes, and representation-level interventions.

Figures & tables

Figure 1: TopK SAE produce $\mathbf{z}_{t}$ on top of a frozen speech encoder, which is routed into linguistic $\mathbf{z}_{L}$ and paralinguistic $\mathbf{z}_{P}$ representations.

			LibriSpeech			MSP-Podcast
Encoder	SAE Model	Route	Recon. $\downarrow$	PER $\downarrow$	SID Acc. $\uparrow$	Recon. $\downarrow$	PER $\downarrow$	SID Acc. $\uparrow$	Emo. UAR $\uparrow$	$F_{0}$ corr. $\uparrow$	Energy corr. $\uparrow$
SPEAR X-Large	Recon-only	$\mathbf{z}_{t}$	0.066	6.00	99.6	0.075	32.37	61.74	65.25	0.578	0.944
	Fixed	$\mathbf{z}_{t}$	0.179	10.50	100.0	0.146	30.52	65.98	66.25	0.482	0.923
		$\mathbf{z}_{L}$		10.02	6.2		30.58	20.67	51.71	0.118	0.781
		$\mathbf{z}_{P}$		99.60	99.8		81.79	76.47	67.40	0.564	0.918
	Quota-freeze	$\mathbf{z}_{t}$	0.163	8.73	100.0	0.171	34.74	68.16	67.46	0.485	0.924

Table 1: Independent probes across encoders and corpora. Reconstruction MSE (Recon.) is reported once per SAE model.

Outcome	Fixed	Freeze
$P$ -swap phone retention	99.57	99.16
$P$ -swap donor-speaker match	98.00	97.20
$L$ -swap phone replacement	94.26	94.12
$L$ -swap recipient-speaker match	96.00	96.80
$L$ -swap donor-phone retention	99.10	98.84

Table 2: Controlled route interventions on the fixed 250-pair SPEAR evaluation set (%).

Routing	Condition	$\mathbf{z}_{L}$ PER	$\mathbf{z}_{P}$ PER	$\mathbf{z}_{L}$ SID	$\mathbf{z}_{P}$ SID
Routing	Condition	(%)	(%)	(%)	(%)
Fixed	Full model	10.02	99.60	6.2	98.6
	No adversaries	4.49	66.64	99.8	100.0
	No speaker adv.	4.63	85.25	100.0	99.4
	No phone adv.	11.01	47.82	1.6	100.0
Freeze	Full model	8.94	80.26	0.6	99.6

Table 3: Ablations of the adversarial objectives on SPEAR. Lower

\mathbf{z}_{L}

SID and higher

\mathbf{z}_{P}

PER indicate stronger cross-factor suppression.

Explore similar work

Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models
Sep 1, 2026Bhuvan Koduru, Dareen Safar B Alharthi, Rita Singh +1Audio-Language Model Evaluation Audio-Language Models
Language Technologies Institute Carnegie Mellon University
Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders
Jun 8, 2026Nikita Koriagin, Georgii Aparin, Nikita Balagansky +1TTS Synthesis Language Model Steering
♣T-Tech · ♦AI Foundation and Algorithm Lab
On the Interpretability of Whisper Encodings Using Sparse Autoencoders
May 12, 2026Dan Pluth, Zachary Nicholas Houghton, Yu Zhou +1Transformer Interpretability Mechanistic Interpretability
Vail Systems, Inc. · University of Oregon

Sep 1, 2026, cs.SDJ/K move · Enter open · S save

Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

Bhuvan Koduru, Dareen Safar B Alharthi, Rita Singh, Bhiksha Raj

Language Technologies Institute Carnegie Mellon University

Audio-Language Model Evaluation Audio-Language Models Mechanistic Interpretability Speech Prosody

Open paper AskPDF arXiv

Audio language models are designed to understand speech, yet it remains unclear whether they capture how something is said beyond what is said. We present a mechanistic analysis of paralinguistic information in four open source models, Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5-Omni-7B, and Chroma-4B, using the Expresso dataset with controlled speaking styles. We combine centered kernel alignment, linear probing with leave one speaker out evaluation, open ended tone prediction, and a content prosody leakage metric to trace how style information moves from the audio encoder to the final output. All models strongly encode speaking style in the late encoder, that is, the top third of the audio encoder's layers, but this information is consistently degraded before reaching the output. The projector reshapes representation geometry without removing information, while decoders differ in how much style they preserve depending on architecture and training objective. At the output level, models fall into two behaviors. Some are content driven, where predictions depend mainly on text. Others are acoustic driven, where predictions vary with speaking style. The leakage metric quantifies this difference, and qualitative results confirm it. Overall, we identify a gap between what models encode and what they use, highlighting a key limitation in current audio language models.

Abstract

Explore similar work

Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders

On the Interpretability of Whisper Encodings Using Sparse Autoencoders