InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech

Sihang Nie; Xueru Li; Xiaofen Xing; Deyi Tuo; Cheng-Bin Jin; Jingyuan Xing; Jinxin Ji

eess.ASSep 28, 2026

InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech

Authors: Sihang Nie, Xueru Li, Xiaofen Xing, Deyi Tuo, Cheng-Bin Jin, Jingyuan Xing, Jinxin Ji

Organizations: South China University of Technology · Huya Inc. · The Hongkong Polytechnic University

Abstract

Instruction-based text-to-speech (ITTS) systems enable natural-language control of expressive speech generation, but often offer limited transparency and fine-grained control over individual text units. Character-level controllable TTS systems provide explicit acoustic control, yet typically rely on user-specified acoustic attributes. To bridge this gap, we propose InstCharVoice, a unified framework that grounds natural-language instructions in character-level acoustic control. We first construct grounded instruction annotations on the WordVoice-5A-zh corpus using Qwen3-Omni. With this supervision, we train an autoregressive model to identify instruction-relevant characters and predict their acoustic attributes before generating the corresponding speech tokens. Keyword prediction and grounding-aware loss weighting help the model focus on instruction-relevant characters and attributes. Experiments show improved instruction following and keyword-level acoustic control over representative ITTS systems, with competitive speech naturalness and explicit character-level controllability. Audio samples are available at https://xxh333.github.io/instcharvoice-demo/.

Figures & tables

Figure 1: Motivation and overview of InstCharVoice. The framework bridges natural-language instructions and explicit character-level acoustic control through instruction grounding and acoustic planning.

Figure 2: Model architecture of InstCharVoice, including (a) training sequence construction, (b) autoregressive modeling of character-level speech chunks, and (c) grounding-aware training objective.

Statistic	Utterances	Keywords	Dur	Bnd	Eng	Pit	Ton
Count	2143k	5978k	609k	3882k	2830k	202k	901k

Table 1: Statistics of the character-level grounded instruction annotations. Dur, Bnd, Eng, Pit, and Ton denote duration, boundary, energy, pitch, and tone, respectively.

Model	DNSMOS $\uparrow$	CER $\downarrow$	kD-MAE $\downarrow$	kE-MAE $\downarrow$	kP-MAE $\downarrow$	kB-RER $\downarrow$	kT-RER $\downarrow$
GroundTruth	3.616	—	0.0182	0.0133	0.0094	5.26%	8.95%
FireRedTTS3	3.599	2.89%	0.1162	0.1409	0.2326	31.29%	47.02%
Qwen3-TTS	3.642	2.99%	0.1057	0.1930	0.2291	28.36%	44.56%
CosyVoice3	3.583	3.12%	0.1126	0.1411	0.2383	24.27%	47.84%
InstCharVoice	3.686	3.58%	0.0769	0.1271	0.1857	14.04%	37.05%
- w/o Instruction	3.560	2.91%	0.0965	0.1431	0.2401	23.10%	44.08%

Table 2: Objective comparison of different systems.

Figure 3: Blind evaluation of instruction annotation consistency.

Model	NMOS $\uparrow$	IMOS $\uparrow$	SMOS $\uparrow$
FireRedTTS3	3.331 ± 0.266	3.036 ± 0.306	3.533 ± 0.212
Qwen3-TTS	3.277 ± 0.276	3.357 ± 0.230	3.124 ± 0.246
CosyVoice3	3.297 ± 0.264	3.402 ± 0.253	3.374 ± 0.227
InstCharVoice	3.528 ± 0.219	3.602 ± 0.212	3.173 ± 0.260

Table 3: Subjective evaluation results.

Model	kD-MAE $\downarrow$	kE-MAE $\downarrow$	kP-MAE $\downarrow$	kB-RER $\downarrow$	kT-RER $\downarrow$
Full	0.0769	0.1271	0.1857	14.04%	37.05%
- w/o KP	0.0805	0.1383	0.1899	16.37%	37.82%
- w/o GLW	0.0859	0.1389	0.2019	20.76%	41.82%
- w/o Both	0.0906	0.1583	0.2132	21.54%	42.58%

Table 4: Ablation study results.

Model	D-MAE $\downarrow$	E-MAE $\downarrow$	P-MAE $\downarrow$	B-ER $\downarrow$	T-RER $\downarrow$
WordVoice	0.0348	0.0484	0.1093	12.72%	24.29%
InstCharVoice	0.0393	0.0544	0.1065	12.14%	21.87%

Table 5: Character-level control results.

Explore similar work

AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech
May 14, 2026Bin Kang, Shaoguo Wen, Yang Fan +6Text-To-Speech Synthesis Acoustic Latent Space
University of Chinese Academy of Sciences · Shenzhen Loop Area Institute · Tencent Turinglab +2
WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS
Jul 7, 2026Sihang Nie, Jinxin Ji, Xiaofen Xing +4Flow-Matching Text-To-Speech Prosody
South China University of Technology · Huya Inc. · Tongji University +2
MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech
Apr 20, 2026Huakang Chen, Jingbin Hu, Liumeng Xue +12Seed-Tts-Eval Benchmark Multilingual Benchmark
School of Intelligence Science and Technology, Nanjing University, Suzhou, China · Yutu Zhineng, Beijing, China · Lingguang Zhaxian Technology, Shanghai, China

May 14, 2026, cs.CVJ/K move · Enter open · S save

AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech

Bin Kang, Shaoguo Wen, Yang Fan, Shunlong Wu+5

University of Chinese Academy of Sciences · Shenzhen Loop Area Institute · Tencent Turinglab+2

Text-To-Speech Synthesis Acoustic Latent Space Steering Cognitive Science

Open paper AskPDF arXiv

While existing text-to-speech (TTS) models exhibit high expressiveness, fine-grained control over composite instructions remains challenging due to the structural mismatch between discrete textual intents and continuous acoustic realizations. Inspired by human cognitive decoupling, we introduce AgentSteerTTS, a multi-agent closed-loop framework designed for intent-faithful expressive control of composite instructions. First, in our framework, an adversarial disentanglement agent mitigates speaker-emotion leakage by learning separable identity and emotion-prosody subspaces with leakage-suppressing regularization. Next, a Dual-Stream Anchoring Controller grounds abstract intents using a large-scale acoustic prototype library: a Retrieval Agent selects expressive anchors, while a Synthesis Agent fuses them into continuous control vectors via gated attention. Finally, a Fast-Slow Feedback Agent refines output intensity through latent gradient correction and resolves semantic-acoustic mismatches using high-level perceptual critique. Experiments on a composite-instruction benchmark and public test sets show that AgentSteerTTS yields consistent and significant improvements to the baselines, demonstrating the effectiveness of the proposed method.

Abstract

Explore similar work

AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech

WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS

MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech