Controllable Speech Generation

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, up 600% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 84

All topics

Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech
May 31, 2026Hongfei Du, Jiacheng Shi, Sidi Lu +2TTS Synthesis Sparse Autoencoders
Department of Computer Science, William & Mary, USA.
Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning
May 30, 2026Sukru Samet Dindar, Riki Shimizu, Xilin Jiang +1Spoken Dialogue Systems Controllable Speech Generation
Department of Electrical Engineering Columbia University New York, NY 10025
HoliTok:A Coutinuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding
May 28, 2026Bohan Li, Shi Lian, Hankun Wang +6Audio Representation Learning Speech Foundation Models
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China · hi lab, Xiaohongshu Inc, China
PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis
May 26, 2026Bowen Li, Shaotong Guo, Zhen Wang +11TTS Synthesis Zero-Shot TTS
Amap, Alibaba Group · The Chinese University of Hong Kong, Shenzhen
Can We Hear from Events? Generating Speech from Event Camera
May 26, 2026Jingping Fang, Lin Chen, Chenyang Xu +3Multimodal Generation Controllable Speech Generation
Beijing Technology and Business University. · Xidian University. · Tongji University. +1
Thinking-while-speaking: A Controlled, Interleaved Reasoning Method for Real-Time Speech Generation
May 20, 2026Xuan Du, Qiangyu Yan, Wenshuo Li +4RL for Language Model Reasoning Controllable Speech Generation
Huawei Technologies
DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech
May 20, 2026Xu Zhang, Longbing Cao, Zhangkai WuTTS Synthesis Audio Diffusion Models
Frontier AI Research Centre, Macquarie University, Sydney, NSW, Australia
Bridging the Gap: Converting Read Text to Conversational Dialogue
May 18, 2026Parshav Singla, Agnik Banerjee, Aaditya Arora +5TTS Synthesis Speech Prosody
Thapar Institute of Engineering and Technology, Patiala, India · Samsung Research and Development Institute, Bangalore, India
Voice "Cloning" is Style Transfer
May 15, 2026Kaitlyn Zhou, Federico Bianchi, Martijn Bartelds +3Controllable Speech Generation Trust in AI
Cornell University · TogetherAI · Stanford University
AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech
May 14, 2026Bin Kang, Shaoguo Wen, Yang Fan +6TTS Synthesis Controllable Speech Generation
University of Chinese Academy of Sciences · Shenzhen Loop Area Institute · Tencent Turinglab +2
VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing
May 7, 2026Jiacheng Xu, Heting Gao, Liufei Xie +8Singing Voice Synthesis Spoken Dialogue Systems
QinYu Team & VITA-Team
SPARCLE: SPeaker-aware Aligned Representations via Contrastive Language Embeddings
May 1, 2026Priyam Mazumdar, Yurii Halychanskyi, Steven Guo +2Representation Learning TTS Synthesis
University of Illinois Urbana-Champaign, USA · National Center for Supercomputing Applications, USA
EmoTransCap: Dataset and Pipeline for Emotion Transition-Aware Speech Captioning in Discourses
Apr 29, 2026Shuhao Xu, Yifan Hu, Jingjing Wu +3Emotion Recognition in Conversations Controllable Speech Generation
Inner Mongolian University, China · State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University
MAGIC-TTS: Fine-Grained Controllable Speech Synthesis with Explicit Local Duration and Pause Control
Apr 23, 2026Jialong Mai, Xiaofen Xing, Xiangmin XuTTS Synthesis Speech Prosody
South China University of Technology
ATRIE: Adaptive Tuning for Robust Inference and Emotion in Persona-Driven Speech Synthesis
Apr 21, 2026Aoduo Li, Haoran Lv, Hongjian Xu +5TTS Synthesis Controllable Speech Generation
Guangdong University of Technology Guangzhou, China · South China University of Technology Guangzhou, China · Shenzhen Polytechnic University Shenzhen, China +2
MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech
Apr 20, 2026Huakang Chen, Jingbin Hu, Liumeng Xue +12TTS Synthesis Speech Generation Evaluation
School of Intelligence Science and Technology, Nanjing University, Suzhou, China · Yutu Zhineng, Beijing, China · Lingguang Zhaxian Technology, Shanghai, China
NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generation
Apr 17, 2026Liumeng Xue, Weizhen Bian, Jiahao Pan +9Speech Generation Evaluation Speech Quality Assessment
Nanjing University · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +4
AST: Adaptive, Seamless, and Training-Free Precise Speech Editing
Apr 17, 2026Sihan Lv, Yechen Jin, Zhen Li +5TTS Synthesis Audio Editing
School of Software Technology, Zhejiang University, Hangzhou, China · Institute of Remote Sensing Satellite, China Academy of Space Technology, Beijing, China · Innovation and Management Center of the School of Software (Ningbo), Zhejiang University, Ningbo, China +1
OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models
Mar 25, 2026Seunghee Kim, Bumkyu Park, Kyudan Jung +5Multimodal Grounding Unified Multimodal Models
Hanyang University · Seoul National University · KAIST AI +1
SelfTTS: cross-speaker style transfer through explicit embedding disentanglement and self-refinement using self-augmentation
Mar 23, 2026Lucas H. Ueda, João G. T. Lima, Pedro R. Corrêa +3Disentangled Representation Learning TTS Synthesis
Universidade Estadual de Campinas (UNICAMP), Brazil · CPQD, Brazil
Zero-Shot Lombard Speech Synthesis with Controllable Style Embeddings
Jan 19, 2026Seymanur Akti, Alexander WaibelTTS Synthesis Zero-Shot TTS
KIT Campus Transfer GmbH (KCT) · Karlsruhe Institute of Technology (KIT) · Carnegie Mellon University (CMU)
CosyEdit: Unlocking End-to-End Speech Editing Capability from Zero-Shot Text-to-Speech Models
Jan 8, 2026Junyang Chen, Yuhang Jia, Hui Wang +2Audio Editing Speech Editing
College of Computer Science, Nankai University, Tianjin, China
PitchFlower: A flow-based neural audio codec with pitch controllability
Oct 29, 2025Diego Torres, Axel Roebel, Nicolas ObinNeural Audio Codecs Controllable Speech Generation
STMS Lab IRCAM - CNRS - Sorbonne Universit´e Paris, France
UtterTune: LoRA-Based Target-Language Pronunciation Edit and Control in Multilingual Text-to-Speech
Aug 13, 2025Shuhei KatoTTS Synthesis Speech Prosody
Independent Researcher Tokyo, Japan

May 31, 2026, cs.CLJ/K move · Enter open · S save

Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech

Hongfei Du, Jiacheng Shi, Sidi Lu, Gang Zhou+1

Department of Computer Science, William & Mary, USA.

TTS Synthesis Sparse Autoencoders Mechanistic Interpretability Controllable Speech Generation Emotional Speech Synthesis

Open paper AskPDF arXiv

Integrating large language models (LLMs) into text-to-speech (TTS) systems has improved speech expressiveness, yet interpretable emotional control remains challenging. Existing approaches primarily rely on external conditioning or global activation steering, offering limited insight into the internal representations underlying emotional control. In this work, we analyze emotion-related variation in the semantic hidden states of LLM-based TTS models using sparse autoencoders (SAEs) to identify sparse latent features. Our analysis shows that emotional variation is distributed across multiple sparse latent features, while intervening on a small subset enables interpretable emotion control. Building on this observation, we introduce a feature-level intervention framework for bidirectional emotion induction and suppression without modifying backbone parameters. We further show that distinct latent features are associated with specific acoustic attributes (e.g., pitch), suggesting that emotional expression arises from coordinated latent contributions rather than a single global shift. Empirically, steering these sparse latent features achieves comparable or superior emotion induction and suppression performance relative to global steering and existing TTS baselines.

Momentum

Related topics

Latest papers 84