High linear decodability does not entail causal controllability in continuous-latent text-to-speech (TTS). In VoxCPM2, position-matched probes and equal-norm interventions across 30 speakers and 60 directed dyads expose this separation: reference-audio states at the final residual acoustic language model layer (RALM L7) retain 0.970 probe accuracy, yet aligned edits yield 0/60 target-dominant dyads. Edits at early reference-audio positions produce the opposite pattern. At L0, pair-aligned updates increase target similarity by 0.326 and achieve target dominance in all 60 dyads, against at most 1/60 under any equal-norm direction or position control. Independent speaker encoders, retrieval against held-out speaker prototypes, and double-blind listening confirm that the shifted voice is heard as the intended target. Cross-architecture interventions in dots.tts distinguish backbone leverage from direct acoustic control: backbone edits displace the source without target attraction, while x-vector ablation reduces source similarity from 0.849 to 0.006. These findings identify two requirements for training-free speaker control: geometric specificity and a computational route that carries the intended speaker condition into acoustic synthesis.
Figures & tables
Figure 1: (a) In VoxCPM2, speaker identity at reference-audio positions remains highly decodable ( > 94%, solid) while final-state decodability rises with depth (dashed). Bottom: 60-dyad target gain at L0/L3/L7 (markers) with target-dominant counts; a finer-grained six-layer sweep over 25 dyads (thin line) resolves a sharp decline in target control at L5. (b) Early reference-audio aligned updates produce the largest target gains among equal-norm direction and position conditions. (c) Held-out-reference listeners confirm perceptual direction and depth selectivity. (d) The dots.tts interventions separate backbone leverage from direct speaker conditioning. At L0, aligned updates produce the largest source displacement and highest WER; mean target gain is non-positive across directions. X-vector ablation reduces source similarity from 0.849 to 0.006. Uncertainty in (b,d) is evaluated by dyad-bootstrap (95% CIs; 60 directed dyads after averaging seeds and prompts within each dyad).
Reference-audio
Final-state
Layer
Probe acc.
Target gain [95% CI]
Dominant
Probe acc.
Target gain [95% CI]
Dominant
L0
0.947
+0.326 [0.298, 0.353]
60/60
0.637
+0.020 [0.006, 0.035]
1/60
L3
0.963
+0.227 [0.204, 0.250]
52/60
0.900
+0.038 [0.021, 0.055]
4/60
L7
0.970
0.000 [ − 0.002, 0.001]
0/60
0.933
+0.042 [0.027, 0.057]
7/60
Table 1: Speaker identity remains decodable across depth, while target control concentrates at early reference-audio positions. Reference-audio probes use the token mean. Target gain is ΔSIMB , with dyad-bootstrap 95% CIs; dominant counts are out of 60 directed dyads. Probe and intervention samples are disjoint and share the same 30-speaker pool.
Held-out target retrieval
Layer
CAM++ gain / dom.
ECAPA
CAM++
Aligned Δ WER
L0
+0.237 / 55/60
46/60
46/60
−0.003[−0.018,0.013]
L3
+0.190 / 43/60
44/60
41/60
−0.003[−0.015,0.007]
L7
+0.001 / 0/60
0/60
0/60
−0.001[−0.003,0.001]
Table 2: Cross-encoder and held-out target evaluation. CAM++ gain denotes aligned ΔSIMB ; dom. counts target-dominant dyads. Retrieval is top-1 over 30 held-out speaker prototypes. CAM++ and retrieval use the first prompt; Whisper ΔWER uses both prompts. All counts are out of 60 dyads; brackets report dyad-bootstrap 95% CIs for ΔWER .
Layer
Direction
Source change ΔSIMA
Target gain ΔSIMB
WER (absolute)
L0
Aligned
−0.197
−0.020
0.475
L0
Orthogonal
−0.032
−0.014
0.107
L0
Random
−0.030
−0.007
0.089
L14
Aligned
−0.187
−0.004
0.421
L14
Orthogonal
−0.049
−0.009
0.095
L14
Random
−0.022
−0.010
0.075
Table 3: Backbone leverage and target specificity in dots.tts under equal-norm direction controls across 60 directed dyads, two prompts, and three seeds. Similarity changes use ECAPA-TDNN and matched baselines; WER is absolute Whisper word-error rate. Every condition has 0/60 target-dominant dyads. Aligned updates show the strongest source-displacement and WER response at each layer; mean target gain is non-positive throughout.
Language models increasingly serve as the backbone of text-to-speech (TTS) systems, yet we understand little about the representations they build when text and generated speech tokens share a single residual stream. We train BatchTopK sparse autoencoders on the LM backbone of CosyVoice3 and introduce a modality-aware auto-interp pipeline that labels each feature from where it fires-text-prefix context, 1-second speech clips, or both. The recovered features are interpretable, spanning phonemes, laughter, accent prompts and speaker gender. Steering through the SAE latent space shows these features are causal rather than merely descriptive: targeted interventions raise laughter probability from 0.02 to 0.79, flip perceived speaker gender, and control speech rate while preserving spoken content. SAE features thus serve both as interpretability objects and as control directions for TTS synthesis.
Nikita Koriagin, Georgii Aparin, Nikita Balagansky +1
Speech editing for content creation requires precise control over both what an edit should do and where it should apply. Free-form natural language provides a flexible interface for expressing edit requests, but its ambiguity may leave the intended operation, parameters, or target region underspecified. We study a precise and explicit interface for speech editing: a transcript-grounded structural edit instruction with XML-style tags explicitly specifies typed operations and localizes them to transcript spans or boundaries. This semantic timeline avoids explicit timestamp alignment and provides an externally inspectable contract for compositional edits. We instantiate the interface in dots.tts.edit, an editor adapted from the continuous autoregressive dots.tts foundation model. Four representative speech-creation controls cover lexical content, affective expression, pitch and speaking-rate delivery, and temporal phrasing through text, emotion, prosody, and pause editing. Task-specific data pipelines construct operation- and scope-controlled pairs while retaining source-derived context outside each target region. We further introduce doteBench, a bilingual evaluation suite that measures precise instruction following, local preservation, and audio quality across the four controls and their composition. Experiments show leading overall instruction following and local preservation across its five editing categories, while audio quality remains comparable to existing open-source systems. Across three Seed-TTS-Eval shards, the model shows negligible differences from the base model in zero-shot TTS recognition error rate and speaker similarity.
Hankun Wang, Bohan Li, Shi Lian +7
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University · Xiaohongshu Inc.
Multi-speaker dialogue TTS requires natural speech generation, consistent speaker identity, coherent cross-turn transitions, and fine-grained control of expressive attributes such as emotion, speaking rate, and loudness. These requirements are difficult to satisfy reliably with one-shot generation, especially in long-form dialogue. We propose a controllable multi-speaker dialogue TTS framework that formulates synthesis as critique-driven iterative refinement. Its speech backbone, ControlEdit-TTS, unifies instruction-following synthesis and natural-language-guided attribute editing, enabling correction of expressive errors without full regeneration. The framework further performs hierarchical utterance-level and scene-level critique, routing detected issues to editing, resynthesis, or timing adjustment. Experiments on a bilingual Chinese--English dialogue benchmark show improved utterance-level instruction following, better dialogue-level preference than direct dialogue models and agentic baselines, and more effective refinement than regeneration-only alternatives while preserving speaker identity. Ablations further confirm the benefits of scene-level critique and edit-based correction.