Controllable Speech Generation

Overview

Systems typically condition autoregressive, diffusion, or flow-matching text-to-speech models on instructions, reference audio, speaker or style embeddings, or explicit phone- and word-level pitch, duration, and energy controls. Other approaches use disentangled latent spaces, task-vector or activation steering, adapters, and reinforcement or preference optimization to edit attributes such as emotion, accent, speaking rate, emphasis, and nonverbal vocalizations. These controls support voice design, speech editing, dubbing, dialogue agents, and intelligibility adjustment; benchmarks such as InstructTTSEval and Seed-TTS-Eval assess instruction following alongside speech quality and speaker similarity. A central challenge is changing the requested attribute while preserving content, identity, naturalness, timing, and other unrequested traits, especially for compositional instructions across speakers and languages.

Latest papers

CardsList

121 papers5 papers in the last four weeks, down 82% on the four weeks before. 0.0% of all new papers.