Controllable Speech Generation

Ask about this topic

Overview

Systems typically condition autoregressive, diffusion, or flow-matching text-to-speech models on instructions, reference audio, speaker or style embeddings, or explicit phone- and word-level pitch, duration, and energy controls. Other approaches use disentangled latent spaces, task-vector or activation steering, adapters, and reinforcement or preference optimization to edit attributes such as emotion, accent, speaking rate, emphasis, and nonverbal vocalizations. These controls support voice design, speech editing, dubbing, dialogue agents, and intelligibility adjustment; benchmarks such as InstructTTSEval and Seed-TTS-Eval assess instruction following alongside speech quality and speaker similarity. A central challenge is changing the requested attribute while preserving content, identity, naturalness, timing, and other unrequested traits, especially for compositional instructions across speakers and languages.

Latest papers

CardsList

117 papers21 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

  1. MAGIC-TTS: Fine-Grained Controllable Speech Synthesis with Explicit Local Duration and Pause Control

    Apr 23, 2026Jialong Mai, Xiaofen Xing, Xiangmin XuTTS SynthesisSpeech Prosody

    South China University of Technology

  2. ATRIE: Adaptive Tuning for Robust Inference and Emotion in Persona-Driven Speech Synthesis

    Apr 21, 2026Aoduo Li, Haoran Lv, Hongjian Xu +5TTS SynthesisControllable Speech Generation

    Guangdong University of Technology Guangzhou, China · South China University of Technology Guangzhou, China · Shenzhen Polytechnic University Shenzhen, China +2

  3. MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech

    Apr 20, 2026Huakang Chen, Jingbin Hu, Liumeng Xue +12TTS SynthesisSpeech Generation Evaluation

    School of Intelligence Science and Technology, Nanjing University, Suzhou, China · Yutu Zhineng, Beijing, China · Lingguang Zhaxian Technology, Shanghai, China

  4. NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generation

    Apr 17, 2026Liumeng Xue, Weizhen Bian, Jiahao Pan +9Nonverbal Vocalization GenerationSpeech Generation Evaluation

    Nanjing University · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +4

  5. AST: Adaptive, Seamless, and Training-Free Precise Speech Editing

    Apr 17, 2026Sihan Lv, Yechen Jin, Zhen Li +5TTS SynthesisAudio Editing

    School of Software Technology, Zhejiang University, Hangzhou, China · Institute of Remote Sensing Satellite, China Academy of Space Technology, Beijing, China · Innovation and Management Center of the School of Software (Ningbo), Zhejiang University, Ningbo, China +1

  6. Bridging What the Model Thinks and How It Speaks: Expressive Speech Generation via Self-Aware Intent-Realization Alignment

    Apr 13, 2026Kuang Wang, Lai Wei, Ping Lin +8Expressive Speech SynthesisSpeech Language Models

    The Chinese University of Hong Kong, Shenzhen · Tencent Ethereal Audio Lab · Southeast University +2

  7. CapTalk: Unified Voice Design for Single-Utterance and Dialogue Speech Generation

    Apr 9, 2026Xiaosu Su, Zihan Sun, Peilei Jia +1TTS SynthesisControllable Speech Generation

    University of Chinese Academy of Sciences Beijing, China · Hello Group Inc. Beijing, China

  8. Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models

    Apr 9, 2026Jaehoon Kang, Yejin Lee, Yoonji Park +1TTS SynthesisControllable Speech Generation

    Department of Artificial Intelligence, Sungkyunkwan University, Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Korea

  9. MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions

    Mar 30, 2026Kexin Huang, Liwei Fan, Botian Jiang +11Controllable Speech GenerationTTS Synthesis

  10. OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models

    Mar 25, 2026Seunghee Kim, Bumkyu Park, Kyudan Jung +5Multimodal GroundingUnified Multimodal Models

    Hanyang University · Seoul National University · KAIST AI +1

  11. SelfTTS: cross-speaker style transfer through explicit embedding disentanglement and self-refinement using self-augmentation

    Mar 23, 2026Lucas H. Ueda, João G. T. Lima, Pedro R. Corrêa +3Disentangled Representation LearningTTS Synthesis

    Universidade Estadual de Campinas (UNICAMP), Brazil · CPQD, Brazil

  12. Gesture2Speech: How Far Can Hand Movements Shape Expressive Speech?

    Mar 20, 2026Lokesh Kumar, Nirmesh Shah, Ashishkumar P. Gudmalwar +1TTS SynthesisControllable Speech Generation

    Media Analysis, Sony Research India

  13. Borderless Long Speech Synthesis

    Mar 20, 2026Xingchen Song, Di Wu, Dinghao Zhou +12Multi-Speaker TTSLong-Form Speech Recognition

    MiLM Plus, Xiaomi Inc., China · WeNet Open Source Community · Nanjing University, China

  14. MOSS-TTS Technical Report

    Mar 18, 2026Yitian Gong, Botian Jiang, Yiwei Zhao +23TTS SynthesisSpeech Foundation Models

  15. Neuron-Level Emotion Control in Speech-Generative Large Audio-Language Models

    Mar 18, 2026Xiutian Zhao, Ismail Rasim Ulgen, Philipp Koehn +2Controllable Speech GenerationEmotional Speech Synthesis

    Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK

  16. CAST-TTS: A Simple Cross-Attention Framework for Unified Timbre Control in TTS

    Mar 17, 2026Zihao Zheng, Wen Wu, Chao Zhang +2TTS SynthesisControllable Speech Generation

    Shanghai AI Lab, China · MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University, China

  17. VoXtream2: Full-stream TTS with dynamic speaking rate control

    Mar 13, 2026Nikita Torgashov, Gustav Eje Henter, Gabriel SkantzeStreaming TTS SynthesisZero-Shot TTS

    Department of Speech, Music and Hearing, KTH Royal Institute of Technology, Sweden

  18. Causal Prosody Mediation for Text-to-Speech:Counterfactual Training of Duration, Pitch, and Energy in FastSpeech2

    Mar 12, 2026Suvendu Sekhar MohantyTTS SynthesisSpeech Prosody

    Arlington, Virginia, USA

  19. Fish Audio S2 Technical Report

    Mar 9, 2026Shijia Liao, Yuxuan Wang, Songting Liu +11Multi-Speaker TTSTTS Synthesis

  20. StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Control

    Mar 8, 2026Haishu Zhao, Aokai Hao, Yuan Ge +3Controllable Speech GenerationSpeech Language Models

    NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China

  21. Accent Vector: Controllable Accent Manipulation for Multilingual TTS Without Accented Data

    Mar 8, 2026Thanathai Lertpetchpun, Thanapat Trachu, Jihwan Lee +3Multilingual TTS SynthesisTTS Synthesis

    Signal Analysis and Interpretation Lab, University of Southern California, USA · Thomas Lord Department of Computer Science, University of Southern California, USA · Department of Linguistics, University of Southern California

  22. CosyAccent: Duration-Controllable Accent Normalization Using Source-Synthesis Training Data

    Feb 22, 2026Qibing Bai, Shuhao Shi, Shuai Wang +3Speech ProcessingControllable Speech Generation

    SDS, The Chinese University of Hong Kong, Shenzhen, China · SRIBD, The Chinese University of Hong Kong, Shenzhen, China · Tencent Ethereal Audio Lab, Tencent, Shenzhen, China +3

  23. Narrating For You: Prompt-guided Audio-visual Narrating Face Generation Employing Multi-entangled Latent Space

    Feb 20, 2026Aashish Chandra, Aashutosh A, Abhijit DasAudio-Video GenerationAudio-Driven Facial Animation

    Machine Intelligence Group, Department of CS&IS, BITS Pilani, Hyderabad Campus, India · Georgia Institute of Technology, USA

  24. Zero-Shot Lombard Speech Synthesis with Controllable Style Embeddings

    Jan 19, 2026Seymanur Akti, Alexander WaibelTTS SynthesisZero-Shot TTS

    KIT Campus Transfer GmbH (KCT) · Karlsruhe Institute of Technology (KIT) · Carnegie Mellon University (CMU)

  25. CosyEdit: Unlocking End-to-End Speech Editing Capability from Zero-Shot Text-to-Speech Models

    Jan 8, 2026Junyang Chen, Yuhang Jia, Hui Wang +2Audio EditingSpeech Editing

    College of Computer Science, Nankai University, Tianjin, China

  26. PitchFlower: A flow-based neural audio codec with pitch controllability

    Oct 29, 2025Diego Torres, Axel Roebel, Nicolas ObinNeural Audio CodecsControllable Speech Generation

    STMS Lab IRCAM - CNRS - Sorbonne Universit´e Paris, France

  27. UtterTune: LoRA-Based Target-Language Pronunciation Edit and Control in Multilingual Text-to-Speech

    Aug 13, 2025Shuhei KatoTTS SynthesisMultilingual TTS Synthesis

    Independent Researcher Tokyo, Japan