eess.ASNov 18, 2026

Location Matters: Layer-wise Prompt Conditioning for Voice--Style Disentanglement in Text-to-Speech

Authors: David Lindevelt, Suzan Verberne, Joost Broekens

Organizations: Leiden University LIACS Leiden, The Netherlands

Abstract

Independent control of voice and emotional style characteristics of speech is a long-standing problem in text-to-speech (TTS) systems. We investigate how the conditioning location of voice and style prompts across TTS decoder layers affects the trade-off between preservation and expressiveness for voice identity and emotional style. We introduce a layer-dependent gated fusion mechanism and systematically study which layers should be influenced by which prompt type. Our findings show an asymmetric relationship. Voice benefits from early conditioning for identity preservation, while style benefits from early conditioning for expressiveness. Our findings provide practical guidance that helps neural network architects make an appropriate trade-off for the architecture of TTS systems according to their use cases.

Figures & tables

Explore similar work

CardsList
  1. Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models

    Apr 9, 2026Jaehoon Kang, Yejin Lee, Yoonji Park +1TTS SynthesisControllable Speech Generation

  2. VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS

    Oct 10, 2026Wenbing Yang, Qihang Lu, Zihan Sun +4TTS SynthesisControllable Speech Generation