Controllable Speech Generation

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, up 600% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 84

All topics

Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech
Oct 8, 2026Shiao Zhu, Lianbo Liu, Sizhen Lyu +3TTS Synthesis Controllable Speech Generation
Institute of Science Tokyo, Tokyo, Japan · Independent Researcher, Tokyo, Japan
Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS
Oct 8, 2026Junchuan Zhao, Chenglin Xu, Wei Zeng +3TTS Synthesis Controllable Speech Generation
National University of Singapore · LIGHTSPEED · Nanyang Technological University +1
Steerspeech: Activation Steering For Emotion Control In Generated Speech
Oct 7, 2026Afsara Benazir, Darius Pétermann, Felix Xiaozhu Lin +1Controllable Speech Generation Emotional Speech Synthesis
University of Virginia · Netflix, Inc.
Steering Follows Geometry, Not Labels: Emotion Directions in a Full-Duplex Speech Model
Oct 6, 2026Pulak KuliControllable Speech Generation Speech Language Models
Independent Researcher
Loud and Clear: Dynamic Activation Steering for Improving Speech Intelligibility in Noisy Environments
Oct 6, 2026Seymanur Akti, Alexander WaibelTTS Synthesis Controllable Speech Generation
Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany · KIT Campus Transfer (KCT), Karlsruhe, Germany · Carnegie Mellon University (CMU), Pittsburgh, USA
Tracing a Sparse Emotion-Control Circuit in LLM-Based Text-to-Speech
Oct 4, 2026Hongfei Du, Jiacheng Shi, Yanfu Zhang +1Mechanistic Interpretability Controllable Speech Generation
Department of Computer Science, William & Mary, Williamsburg, USA
SCIC: Scope- and Codebook-Aware Instruction Conditioning for Speaker-Adapted Expressive TTS
Sep 30, 2026Longyu Lu, Zongwei Du, Mengtao Xing +4TTS Synthesis Streaming TTS Synthesis
TaoLive-AIGC Team Taobao & Tmall Group of Alibaba
EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation
Sep 29, 2026Kuan-Po Huang, Haohe Liu, Puyuan Peng +5Controllable Speech Generation Emotional Speech Synthesis
Reality Labs at Meta · National Taiwan University · FAIR at Meta
InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech
Sep 28, 2026Sihang Nie, Xueru Li, Xiaofen Xing +4TTS Synthesis Language Model-Based Control
South China University of Technology · Huya Inc. · The Hongkong Polytechnic University
SEmoEdit: Probing and Harnessing the Editability of Pre-trained Speech Flows
Sep 28, 2026Tianxin Xie, Pengfei Zhang, Kai Jiang +2TTS Synthesis Speech Editing
The Hong Kong University of Science and Technology (Guangzhou)
Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: Exploration toward Age, Gender, and Accent Steering
Sep 28, 2026Shih-Heng Wang, Tiantian Feng, Aditya Kommineni +4Neural Audio Codecs Sparse Autoencoders
University of Southern California, USA
Controlling Speaking Rate in Autoregressive TTS via Activation Steering
Sep 27, 2026Francesco Verdini, Antonis Asonitis, Aref Farhadipour +4TTS Synthesis Controllable Speech Generation
AGIGO · Sapienza University of Rome · ETH Zurich +1
From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS
Sep 27, 2026Kangxiang Xia, Xinfa Zhu, HangRui Hu +8TTS Synthesis Speech Editing
Independent Researcher
ReaFlow-TTS: Realization-Conditioned Flow Matching for High-Quality and Controllable Speech Synthesis
Sep 24, 2026Junyi Zhao, Yihao Qin, Changsheng MaFlow Matching TTS Synthesis
School of Information Science and Technology, Lanzhou University, China
EmphTTS: an emphasis-control TTS with reinforcement learning
Sep 23, 2026Zirui Li, Rech Silas, Lauri Juvela +2TTS Synthesis Controllable Speech Generation
Department of Information and Communications Engineering, Aalto University, Espoo, Finland
Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbing
Sep 22, 2026Alejandro Pérez-González-de-Martos, Florian Lux, Angelina Elizarova +3TTS Synthesis Audio-Visual Synchronization
AppTek GmbH, Germany
Interactive TTS: Dynamic Speaking Style Adaptation for Expressive Speech Synthesis
Sep 22, 2026Wenjie Tian, Kangxiang Xia, Jingbin Hu +7TTS Synthesis Direct Preference Optimization
Alibaba ATH Token Foundry
Learnable Classifier-Free Guidance Null Embeddings for Enhanced Controllable Speech Synthesis
Sep 21, 2026Biel Tura Vecino, Yoach Lacombe, Julian Weber +4Classifier-Free Guidance TTS Synthesis
Cantina Labs
CycleSpeech: Reciprocal Alignment for Instruction-Controlled Speech Synthesis and Paralinguistic Understanding
Sep 21, 2026Huan Liao, Haonan Han, Xingwen Han +3TTS Synthesis Controllable Speech Generation
The Chinese University of Hong Kong, Shenzhen · Tsinghua University · Amphion Technology Co., Ltd.
StepAudio 3 Gen Technical Report
Sep 14, 2026Bin Lin, Bo Zhao, Boyang Wang +68TTS Synthesis Zero-Shot TTS
Post-Training Zero-Shot TTS for Fine-Grained Emotion and Duration Control via Natural Language
Sep 10, 2026Lianru Gao, Yujie Guo, Yong QinTTS Synthesis Zero-Shot TTS
TMCC, College of Computer Science, Nankai University, Tianjin, China
Preference Optimization with LALM Feedback for Continuous Autoregressive Non-Verbal Vocalization Generation
Sep 10, 2026Jingbin Hu, Qirui Zhan, Yuang Cao +7Direct Preference Optimization Controllable Speech Generation
Shenzhen Pimei Technology Co., Ltd., Guangdong, China · Yutu Zhineng, Beijing, China · State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China +1
TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction
Sep 8, 2026Yi-Chang Chen, Chun Wei Chen, Dien-Ruei Wu +7TTS Synthesis Streaming TTS Synthesis
MediaTek Research · Internship at MediaTek Research · National Taiwan University +2
AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
Sep 8, 2026Ziyang Ma, Zhikang Niu, Wenming Tu +30Speech Foundation Models Speech Editing
Stabilizing Instruction Supervision for Instruct-TTS via Controllable Diversification and Drift Filtering
Sep 8, 2026Yizhong Geng, Kecan Mao, Qifei Li +6TTS Evaluation Training Data Curation
Beijing University of Posts and Telecommunications, China · Li Auto, China
KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interaction
Sep 7, 2026Ryuichiro Higashinaka, Shinnosuke Takamichi, Tetsuji OgawaTTS Synthesis Controllable Speech Generation
Nagoya University, Japan. · National Institute of Informatics, LLMC, Japan. · Keio University, Japan. +1
VoxReason: Auditing Source-Grounded Speech Plans Before Synthesis
Sep 2, 2026Mengzhe GengSpeech Generation Evaluation Controllable Speech Generation
National Research Council Canada
Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS
Aug 31, 2026Yan Zhou, Yun Hong, Yang FengControllable Speech Generation Emotional Speech Synthesis
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation
Aug 12, 2026Haowei Lou, Hye-Young Paik, Dai Jia +2Singing Voice Synthesis TTS Synthesis
UNSW Sydney · Dolby Laboratories
CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis
Aug 8, 2026Zhisheng Zheng, Xiaohang Sun, Zhu Liu +5TTS Synthesis Zero-Shot TTS
The University of Texas at Austin · Amazon

Oct 8, 2026, cs.SDJ/K move · Enter open · S save

Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech

Shiao Zhu, Lianbo Liu, Sizhen Lyu, Yuzhe Wang+2

Institute of Science Tokyo, Tokyo, Japan · Independent Researcher, Tokyo, Japan

TTS Synthesis Controllable Speech Generation

Open paper AskPDF arXiv

Natural-language style descriptions provide an interpretable interface between large language models (LLMs) and controllable text-to-speech (TTS). However, using descriptions as pseudo-labels compresses target acoustics into text, and descriptive fidelity need not imply effective control of a particular synthesizer. We empirically show that speech-text alignment only weakly predicts downstream acoustic similarity among candidate instructions for the same utterance. We therefore propose Speech-Rewarded Style Planning (SRSP), which trains a text-based style planner through a frozen downstream TTS model. Given dialogue history and response text, the planner generates candidate instructions and is optimized with group-relative policy optimization (GRPO), using the teacher-forced likelihood of target speech tokens as the reward. On an English subset of the ISCSLP 2026 CoT-TTS corpus, SRSP achieves higher speech-style and emotion similarity to target speech and lower mel-cepstral distortion than the Base LLM and target-audio-informed captioning baselines. LLM-based expressive speech evaluation further shows gains over all baselines in contextual appropriateness and reference consistency.

Momentum

Related topics

Latest papers 84