Controllable Speech Generation

Overview

Systems typically condition autoregressive, diffusion, or flow-matching text-to-speech models on instructions, reference audio, speaker or style embeddings, or explicit phone- and word-level pitch, duration, and energy controls. Other approaches use disentangled latent spaces, task-vector or activation steering, adapters, and reinforcement or preference optimization to edit attributes such as emotion, accent, speaking rate, emphasis, and nonverbal vocalizations. These controls support voice design, speech editing, dubbing, dialogue agents, and intelligibility adjustment; benchmarks such as InstructTTSEval and Seed-TTS-Eval assess instruction following alongside speech quality and speaker similarity. A central challenge is changing the requested attribute while preserving content, identity, naturalness, timing, and other unrequested traits, especially for compositional instructions across speakers and languages.

Latest papers

121 papers5 papers in the last four weeks, down 82% on the four weeks before. 0.0% of all new papers.

Aug 13, 2025

UtterTune: LoRA-Based Target-Language Pronunciation Edit and Control in Multilingual Text-to-Speech

We propose UtterTune, a lightweight method for adapting a multilingual text-to-speech (TTS) system built on a large language model (LLM). It improves control of pronunciation in the target language while preserving performance in the others. Although LLM architectures have enabled TTS models to achieve remarkable naturalness, accurately modeling grapheme-to-phoneme (G2P) mapping and prosody remains challenging, especially when the model omits an explicit G2P module and directly processes minimally encoded text (e.g., byte-pair encoding). UtterTune leverages low-rank adaptation to enable the control of segmental pronunciation and pitch accent at the phoneme level for Japanese speech, the target language in this paper, while maintaining naturalness and speaker similarity in a zero-shot setting. Objective and subjective evaluations confirm its effectiveness.