Decodability Is Not Controllability: Local and Anisotropic Speaker Control in Continuous-Latent TTS

Yuanzhong Zheng; Wei Huang; Ming Ma; Kailun Jin; Youhai Jiang

cs.SDOct 21, 2026

Decodability Is Not Controllability: Local and Anisotropic Speaker Control in Continuous-Latent TTS

Authors: Yuanzhong Zheng, Wei Huang, Ming Ma, Kailun Jin, Youhai Jiang

Organizations: VoxNexus, Singapore

Abstract

High linear decodability does not entail causal controllability in continuous-latent text-to-speech (TTS). In VoxCPM2, position-matched probes and equal-norm interventions across 30 speakers and 60 directed dyads expose this separation: reference-audio states at the final residual acoustic language model layer (RALM L7) retain 0.970 probe accuracy, yet aligned edits yield 0/60 target-dominant dyads. Edits at early reference-audio positions produce the opposite pattern. At L0, pair-aligned updates increase target similarity by 0.326 and achieve target dominance in all 60 dyads, against at most 1/60 under any equal-norm direction or position control. Independent speaker encoders, retrieval against held-out speaker prototypes, and double-blind listening confirm that the shifted voice is heard as the intended target. Cross-architecture interventions in dots.tts distinguish backbone leverage from direct acoustic control: backbone edits displace the source without target attraction, while x-vector ablation reduces source similarity from 0.849 to 0.006. These findings identify two requirements for training-free speaker control: geometric specificity and a computational route that carries the intended speaker condition into acoustic synthesis.

Figures & tables

Figure 1: (a) In VoxCPM2, speaker identity at reference-audio positions remains highly decodable (

>

94%, solid) while final-state decodability rises with depth (dashed). Bottom: 60-dyad target gain at L0/L3/L7 (markers) with target-dominant counts; a finer-grained six-layer sweep over 25 dyads (thin line) resolves a sharp decline in target control at L5. (b) Early reference-audio aligned updates produce the largest target gains among equal-norm direction and position conditions. (c) Held-out-reference listeners confirm perceptual direction and depth selectivity. (d) The dots.tts interventions separate backbone leverage from direct speaker conditioning. At L0, aligned updates produce the largest source displacement and highest WER; mean target gain is non-positive across directions. X-vector ablation reduces source similarity from 0.849 to 0.006. Uncertainty in (b,d) is evaluated by dyad-bootstrap (95% CIs; 60 directed dyads after averaging seeds and prompts within each dyad).

	Reference-audio			Final-state
Layer	Probe acc.	Target gain [95% CI]	Dominant	Probe acc.	Target gain [95% CI]	Dominant
L0	0.947	+0.326 [0.298, 0.353]	60/60	0.637	+0.020 [0.006, 0.035]	1/60
L3	0.963	+0.227 [0.204, 0.250]	52/60	0.900	+0.038 [0.021, 0.055]	4/60
L7	0.970	0.000 [ $-$ 0.002, 0.001]	0/60	0.933	+0.042 [0.027, 0.057]	7/60

Table 1: Speaker identity remains decodable across depth, while target control concentrates at early reference-audio positions. Reference-audio probes use the token mean. Target gain is

\Delta\mathrm{SIM}_{B}

, with dyad-bootstrap 95% CIs; dominant counts are out of 60 directed dyads. Probe and intervention samples are disjoint and share the same 30-speaker pool.

Layer	CAM++ gain / dom.	ECAPA	CAM++	Aligned $\Delta$ WER
		Held-out target retrieval
L0	+0.237 / 55/60	46/60	46/60	$-0.003$ $[-0.018,0.013]$
L3	+0.190 / 43/60	44/60	41/60	$-0.003$ $[-0.015,0.007]$
L7	+0.001 / 0/60	0/60	0/60	$-0.001$ $[-0.003,0.001]$

Table 2: Cross-encoder and held-out target evaluation. CAM++ gain denotes aligned

\Delta\mathrm{SIM}_{B}

; dom. counts target-dominant dyads. Retrieval is top-1 over 30 held-out speaker prototypes. CAM++ and retrieval use the first prompt; Whisper

\Delta\mathrm{WER}

uses both prompts. All counts are out of 60 dyads; brackets report dyad-bootstrap 95% CIs for

\Delta\mathrm{WER}

Layer	Direction	Source change $\Delta\mathrm{SIM}_{A}$	Target gain $\Delta\mathrm{SIM}_{B}$	WER (absolute)
L0	Aligned	$\mathbf{-0.197}$	$-0.020$	$\mathbf{0.475}$
L0	Orthogonal	$-0.032$	$-0.014$	0.107
L0	Random	$-0.030$	$-0.007$	0.089
L14	Aligned	$\mathbf{-0.187}$	$-0.004$	$\mathbf{0.421}$
L14	Orthogonal	$-0.049$	$-0.009$	0.095
L14	Random	$-0.022$	$-0.010$	0.075

Table 3: Backbone leverage and target specificity in dots.tts under equal-norm direction controls across 60 directed dyads, two prompts, and three seeds. Similarity changes use ECAPA-TDNN and matched baselines; WER is absolute Whisper word-error rate. Every condition has 0/60 target-dominant dyads. Aligned updates show the strongest source-displacement and WER response at each layer; mean target gain is non-positive throughout.

Controllable Speech Generation

Controllable speech generation steers synthesized speech's voice, prosody, emotion, timing, pronunciation, and conversational delivery.

See all 117 papers

Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders
Jun 8, 2026Nikita Koriagin, Georgii Aparin, Nikita Balagansky +1TTS Synthesis Language Model Steering
♣T-Tech · ♦AI Foundation and Algorithm Lab
dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model
Aug 2, 2026Hankun Wang, Bohan Li, Shi Lian +7Speech Foundation Models Speech Editing
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University · Xiaohongshu Inc.
From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS
Sep 27, 2026Kangxiang Xia, Xinfa Zhu, HangRui Hu +8TTS Synthesis Speech Editing
Independent Researcher

Jun 8, 2026J/K move · Enter open · S save

Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders

Nikita Koriagin, Georgii Aparin, Nikita Balagansky, Daniil Gavrilov

♣T-Tech · ♦AI Foundation and Algorithm Lab

TTS Synthesis Language Model Steering LLM Interpretability Sparse Autoencoders Controllable Speech Generation Speech Language Models

Open paper AskPDF arXiv

Language models increasingly serve as the backbone of text-to-speech (TTS) systems, yet we understand little about the representations they build when text and generated speech tokens share a single residual stream. We train BatchTopK sparse autoencoders on the LM backbone of CosyVoice3 and introduce a modality-aware auto-interp pipeline that labels each feature from where it fires-text-prefix context, 1-second speech clips, or both. The recovered features are interpretable, spanning phonemes, laughter, accent prompts and speaker gender. Steering through the SAE latent space shows these features are causal rather than merely descriptive: targeted interventions raise laughter probability from 0.02 to 0.79, flip perceived speaker gender, and control speech rate while preserving spoken content. SAE features thus serve both as interpretability objects and as control directions for TTS synthesis.

Abstract

Controllable Speech Generation

Explore similar work

Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders

dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model

From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS