Controllable Speech Generation
Overview
Systems typically condition autoregressive, diffusion, or flow-matching text-to-speech models on instructions, reference audio, speaker or style embeddings, or explicit phone- and word-level pitch, duration, and energy controls. Other approaches use disentangled latent spaces, task-vector or activation steering, adapters, and reinforcement or preference optimization to edit attributes such as emotion, accent, speaking rate, emphasis, and nonverbal vocalizations. These controls support voice design, speech editing, dubbing, dialogue agents, and intelligibility adjustment; benchmarks such as InstructTTSEval and Seed-TTS-Eval assess instruction following alongside speech quality and speaker similarity. A central challenge is changing the requested attribute while preserving content, identity, naturalness, timing, and other unrequested traits, especially for compositional instructions across speakers and languages.
Latest papers
117 papers21 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.