eess.ASOct 29, 2026

Discovering Interpretable Acoustic Features in Autoregressive Voice-Cloning TTS Models

Authors: Anuj Diwan, David Harwath, Eunsol Choi

Organizations: University of Texas at Austin · New York University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Voice-cloning TTS models synthesize an unseen speaker's voice from brief speech prompts, but it remains unclear which acoustic properties they represent and use. We introduce a feature-based interpretability framework for autoregressive voice-cloning TTS models, analyzing features from sparse autoencoders (SAEs), PCA, and random directions via sparse probing, steering, and quantitative auto-interpretability. We focus on Qwen3-TTS and extend probing and steering to 55 additional models to test generalization across architectures. These models encode concepts like pitch, energy, noise, and accent, but sparse prediction and steering are effective only for some concepts. Strong probing does not always translate to steering or vice-versa. SAEs are useful for sparse probing of human-labelled concepts and novel-concept discovery, while PCA is competitive for dense probing and steering. Auto-interpretability finds 684684 automatically validated SAE features in Qwen3-TTS covering speaker clusters, speaking style, and recording conditions beyond known concepts. Overall, we perform a comprehensive interpretability study of autoregressive voice-cloning TTS models.

Figures & tables

Figure 1: Our main interpretability evaluation pipeline. We evaluate feature sets via known-concept probing and steering across 66 voice-cloning TTS models, and through quantitative auto-interpretability in Qwen3-TTS for novel concept discovery.
Figure 2: Feature extraction from TTS residual-stream activations using SAE, PCA, random directions, and untrained SAEs. We study multiple layers in Qwen3-TTS and one in other models.
Figure 3: All-feature and top- 22 nF1. (a) Top 1010 extractor and human-labelled concepts by SAE recoverability in Qwen3-TTS. Takeaway : Many concepts, especially extractor concepts, are recoverable but few are sparsely predicted from 22 features. (b) Mean nF1 over 5555 extractor and 2121 human-labelled concepts across 66 models (layers in Section 2.1 ). Full lists in Appendix E.1 . Takeaway : PCA and random directions have higher all-feature extractor nF1 than SAE, while SAE has higher top- 22 human-labelled nF1 than PCA across all six models.
Figure 4: Steering at λ=1\lambda=1 , averaged across 5555 extractor concepts on 100100 examples. Higgs TTS 3, where SAE steering severely degraded speech, is shown in Appendix C.3 . Takeaway : In Qwen3-TTS and Fish, SAE target changes are competitive with PCA and supervised DiffMean, with smaller off-target changes than PCA; DiffMean has smaller UTMOS losses. Mean target changes are smaller in Higgs TTS 2 and CosyVoice2.
Figure 5: Top 1010 Qwen3-TTS layer- 1919 concepts by SAE median Target Change, comparing all four methods at λ=1\lambda=1 on 100100 examples. Takeaway : SAE target changes are largest for energy, pitch, spectral and voice-quality properties. Quality losses partly reflect deliberately reducing SNR or pushing RMS toward either extreme.
Figure 6: Qwen3-TTS strength sweeps, averaged across a fixed set of 99 probing-selected targets on the same 100100 utterances (Appendix C.2 ). Layers 1919 / 2626 use solid/dotted lines. Takeaway : SAE, PCA, and DiffMean increase target change with strength while random directions remain ineffective. SAE has lower off-target change than PCA but pays a WER/UTMOS cost.
Feature SetCoherenceLabel FitLabel Stab.Valid.
Rand. Guess0.09±0.030.09{\scriptscriptstyle\pm 0.03}0.50±0.040.50{\scriptscriptstyle\pm 0.04}0.38±0.040.38{\scriptscriptstyle\pm 0.04}–
Rand. Dir.0.25±0.020.25{\scriptscriptstyle\pm 0.02}0.60±0.010.60{\scriptscriptstyle\pm 0.01}0.32±0.020.32{\scriptscriptstyle\pm 0.02}1717
PCA0.08±0.010.08{\scriptscriptstyle\pm 0.01}0.50±0.010.50{\scriptscriptstyle\pm 0.01}0.22±0.020.22{\scriptscriptstyle\pm 0.02}11
SAE0.43±0.03\mathbf{0.43}{\scriptscriptstyle\pm 0.03}0.63±0.01\mathbf{0.63}{\scriptscriptstyle\pm 0.01}0.41±0.03\mathbf{0.41}{\scriptscriptstyle\pm 0.03}92\mathbf{92}
↪\hookrightarrow shuf. labels–0.50±0.010.50{\scriptscriptstyle\pm 0.01}––
↪\hookrightarrow rand. audio0.10±0.010.10{\scriptscriptstyle\pm 0.01}0.50±0.010.50{\scriptscriptstyle\pm 0.01}––
Table 1: Auto-interpretability of 500500 Qwen3-TTS layer- 1919 features per set ( 95%95\% bootstrap intervals). Takeaway : SAE scores highest on all three metrics and yields the most validated features.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Table 8
Figure 7: EV-vs- KK curves at epoch 1010 for layers 33 , 99 (top), 1414 , 1919 (middle), and 2626 (bottom). Lines vary MM ; point colors indicate DFF. We choose widths with DFF roughly below 0.150.15 across K∈{32,64,96}K\in\{32,64,96\} , except at layer 33 , and select KK near the curve’s shoulder.
FamilyExtractorScalar outputs used
Pitch and voicingPENN ( Morrison et al., 2023 )f0 mean , voiced fraction , mean voiced run sec , f0 slope
Energy and spectrumlibrosa RMS and spectral centroidrms mean , rms slope , spectral centroid mean
TimingWebRTC VAD, g2p speaking rate, manifest durationpause fraction , mean pause sec , speaking rate , duration
Recording qualityBrouhaha ( Lavechin et al., 2023 ) ; librosa STFT spectral tiltsnr , c50 , spectral tilt
Voice qualityPraat/Parselmouth ( Jadoul et al., 2018 )jitter local , shimmer local , hnr db
FormantsPraat/Parselmouthf1 median , f2 median , f3 median
Appendix
Table 5: Automatic concept extractors. Each scalar yields high- and low-tail concepts except emotion2vec and accent scores, which yield only high-tail concepts, giving 5555 targets in total.
More in this topic

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 394 papers

Explore similar work

CardsList
Jun 8, 2026

Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders

Language models increasingly serve as the backbone of text-to-speech (TTS) systems, yet we understand little about the representations they build when text and generated speech tokens share a single residual stream. We train BatchTopK sparse autoencoders on the LM backbone of CosyVoice3 and introduce a modality-aware auto-interp pipeline that labels each feature from where it fires-text-prefix context, 1-second speech clips, or both. The recovered features are interpretable, spanning phonemes, laughter, accent prompts and speaker gender. Steering through the SAE latent space shows these features are causal rather than merely descriptive: targeted interventions raise laughter probability from 0.02 to 0.79, flip perceived speaker gender, and control speech rate while preserving spoken content. SAE features thus serve both as interpretability objects and as control directions for TTS synthesis.

Nikita Koriagin, Georgii Aparin, Nikita Balagansky +1

♣T-Tech · ♦AI Foundation and Algorithm Lab

TTS SynthesisLanguage Model SteeringLLM InterpretabilitySparse AutoencodersControllable Speech GenerationSpeech Language Models
AskPDF
Oct 16, 2026

Beyond the Benchmark: Conditioning, Domains and Failure Modes of Qwen3-TTS

Qwen3-TTS reports state-of-the-art zero-shot voice cloning, but its technical report leaves open how the model should be conditioned, how it behaves outside read speech, and how it fails. We characterise its two cloning interfaces, in-context learning (ICL) and X-Vector, against Chatterbox, XTTS-v2 and F5-TTS on read, meeting, emotional and air-traffic-control (ATC) speech, using eighteen objective measures, an LLM judge, a spoofing countermeasure and a listening test. Neither interface dominates: ICL clones timbre more closely from correctly transcribed references, X-Vector clones timing, and ICL degrades sharply when a short reference transcript is wrong. Domain explains four times more variance than system choice. Qwen3-TTS leads on intelligibility and pitch register but trails on prosody, and the objective measures rank systems differently from listeners. An off-the-shelf countermeasure flags its clones no more often than genuine recordings, and on ATC radio it fails for every system. Code and audio are released.

Séverin Baroudi, Yanis Labrak, Petr Motlicek +2

Univ Toulon, Aix Marseille Univ, LIS, CNRS, France · Mila – Quebec AI Institute · Idiap Research Institute, Martigny, Switzerland +3

TTS SynthesisVoice CloningSpeech Generation EvaluationSpeech Quality Assessment
AskPDF
May 15, 2026

Voice "Cloning" is Style Transfer

Artificially generated speech is increasingly embedded in everyday life. Voice cloning in particular enables applications where identity preservation is important, such as completing a recording, dubbing in a new language, or preserving the voices of individuals with speech loss. However, in our work, we find that despite the term, voice cloning does not faithfully ''clone'' an individual's voice. Instead, we find that widely-used voice cloning models systematically apply style transfer to source voices. As rated by human annotators, cloned voices are perceived as more authoritative, warm, customer-service-like, and human-like compared to their sources. Human annotators also report greater trust in cloned voices than source voices, and a greater willingness to disclose sensitive personal information to them. Our work furthermore shows that voice cloning leads to homogenization of speaker characteristics, as measured by reduced variance in accent, speaking rate, and the audio embedding space. Together, our results highlight a new set of limitations and risks of voice cloning technology and their potential impact on human behavior.

Kaitlyn Zhou, Federico Bianchi, Martijn Bartelds +3

Cornell University · TogetherAI · Stanford University

Controllable Speech GenerationTrust in AIAudio GenerationVoice Cloning
AskPDF