eess.ASOct 29, 2026

Discovering Interpretable Acoustic Features in Autoregressive Voice-Cloning TTS Models

Authors: Anuj Diwan, David Harwath, Eunsol Choi

Organizations: University of Texas at Austin · New York University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Voice-cloning TTS models synthesize an unseen speaker's voice from brief speech prompts, but it remains unclear which acoustic properties they represent and use. We introduce a feature-based interpretability framework for autoregressive voice-cloning TTS models, analyzing features from sparse autoencoders (SAEs), PCA, and random directions via sparse probing, steering, and quantitative auto-interpretability. We focus on Qwen3-TTS and extend probing and steering to 55 additional models to test generalization across architectures. These models encode concepts like pitch, energy, noise, and accent, but sparse prediction and steering are effective only for some concepts. Strong probing does not always translate to steering or vice-versa. SAEs are useful for sparse probing of human-labelled concepts and novel-concept discovery, while PCA is competitive for dense probing and steering. Auto-interpretability finds 684684 automatically validated SAE features in Qwen3-TTS covering speaker clusters, speaking style, and recording conditions beyond known concepts. Overall, we perform a comprehensive interpretability study of autoregressive voice-cloning TTS models.

Figures & tables

Figure 1: Our main interpretability evaluation pipeline. We evaluate feature sets via known-concept probing and steering across 66 voice-cloning TTS models, and through quantitative auto-interpretability in Qwen3-TTS for novel concept discovery.
Figure 2: Feature extraction from TTS residual-stream activations using SAE, PCA, random directions, and untrained SAEs. We study multiple layers in Qwen3-TTS and one in other models.
Figure 3: All-feature and top- 22 nF1. (a) Top 1010 extractor and human-labelled concepts by SAE recoverability in Qwen3-TTS. Takeaway : Many concepts, especially extractor concepts, are recoverable but few are sparsely predicted from 22 features. (b) Mean nF1 over 5555 extractor and 2121 human-labelled concepts across 66 models (layers in Section 2.1 ). Full lists in Appendix E.1 . Takeaway : PCA and random directions have higher all-feature extractor nF1 than SAE, while SAE has higher top- 22 human-labelled nF1 than PCA across all six models.
Figure 4: Steering at λ=1\lambda=1 , averaged across 5555 extractor concepts on 100100 examples. Higgs TTS 3, where SAE steering severely degraded speech, is shown in Appendix C.3 . Takeaway : In Qwen3-TTS and Fish, SAE target changes are competitive with PCA and supervised DiffMean, with smaller off-target changes than PCA; DiffMean has smaller UTMOS losses. Mean target changes are smaller in Higgs TTS 2 and CosyVoice2.
Figure 5: Top 1010 Qwen3-TTS layer- 1919 concepts by SAE median Target Change, comparing all four methods at λ=1\lambda=1 on 100100 examples. Takeaway : SAE target changes are largest for energy, pitch, spectral and voice-quality properties. Quality losses partly reflect deliberately reducing SNR or pushing RMS toward either extreme.
Figure 6: Qwen3-TTS strength sweeps, averaged across a fixed set of 99 probing-selected targets on the same 100100 utterances (Appendix C.2 ). Layers 1919 / 2626 use solid/dotted lines. Takeaway : SAE, PCA, and DiffMean increase target change with strength while random directions remain ineffective. SAE has lower off-target change than PCA but pays a WER/UTMOS cost.
Feature SetCoherenceLabel FitLabel Stab.Valid.
Rand. Guess0.09±0.030.09{\scriptscriptstyle\pm 0.03}0.50±0.040.50{\scriptscriptstyle\pm 0.04}0.38±0.040.38{\scriptscriptstyle\pm 0.04}–
Rand. Dir.0.25±0.020.25{\scriptscriptstyle\pm 0.02}0.60±0.010.60{\scriptscriptstyle\pm 0.01}0.32±0.020.32{\scriptscriptstyle\pm 0.02}1717
PCA0.08±0.010.08{\scriptscriptstyle\pm 0.01}0.50±0.010.50{\scriptscriptstyle\pm 0.01}0.22±0.020.22{\scriptscriptstyle\pm 0.02}11
SAE0.43±0.03\mathbf{0.43}{\scriptscriptstyle\pm 0.03}0.63±0.01\mathbf{0.63}{\scriptscriptstyle\pm 0.01}0.41±0.03\mathbf{0.41}{\scriptscriptstyle\pm 0.03}92\mathbf{92}
↪\hookrightarrow shuf. labels–0.50±0.010.50{\scriptscriptstyle\pm 0.01}––
↪\hookrightarrow rand. audio0.10±0.010.10{\scriptscriptstyle\pm 0.01}0.50±0.010.50{\scriptscriptstyle\pm 0.01}––
Table 1: Auto-interpretability of 500500 Qwen3-TTS layer- 1919 features per set ( 95%95\% bootstrap intervals). Takeaway : SAE scores highest on all three metrics and yields the most validated features.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Table 8
Figure 7: EV-vs- KK curves at epoch 1010 for layers 33 , 99 (top), 1414 , 1919 (middle), and 2626 (bottom). Lines vary MM ; point colors indicate DFF. We choose widths with DFF roughly below 0.150.15 across K∈{32,64,96}K\in\{32,64,96\} , except at layer 33 , and select KK near the curve’s shoulder.
FamilyExtractorScalar outputs used
Pitch and voicingPENN ( Morrison et al., 2023 )f0 mean , voiced fraction , mean voiced run sec , f0 slope
Energy and spectrumlibrosa RMS and spectral centroidrms mean , rms slope , spectral centroid mean
TimingWebRTC VAD, g2p speaking rate, manifest durationpause fraction , mean pause sec , speaking rate , duration
Recording qualityBrouhaha ( Lavechin et al., 2023 ) ; librosa STFT spectral tiltsnr , c50 , spectral tilt
Voice qualityPraat/Parselmouth ( Jadoul et al., 2018 )jitter local , shimmer local , hnr db
FormantsPraat/Parselmouthf1 median , f2 median , f3 median
Appendix
Table 5: Automatic concept extractors. Each scalar yields high- and low-tail concepts except emotion2vec and accent scores, which yield only high-tail concepts, giving 5555 targets in total.
More in this topic

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 394 papers

Explore similar work

CardsList

  1. Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders

    Jun 8, 2026Nikita Koriagin, Georgii Aparin, Nikita Balagansky +1TTS SynthesisLanguage Model Steering

    ♣T-Tech · ♦AI Foundation and Algorithm Lab

  2. Beyond the Benchmark: Conditioning, Domains and Failure Modes of Qwen3-TTS

    Oct 16, 2026Séverin Baroudi, Yanis Labrak, Petr Motlicek +2TTS SynthesisVoice Cloning

    Univ Toulon, Aix Marseille Univ, LIS, CNRS, France · Mila – Quebec AI Institute · Idiap Research Institute, Martigny, Switzerland +3

  3. Voice "Cloning" is Style Transfer

    May 15, 2026Kaitlyn Zhou, Federico Bianchi, Martijn Bartelds +3Controllable Speech GenerationTrust in AI

    Cornell University · TogetherAI · Stanford University