SCIC: Scope- and Codebook-Aware Instruction Conditioning for Speaker-Adapted Expressive TTS

Longyu Lu; Zongwei Du; Mengtao Xing; Zhuoqun Liu; Zifan Guan; Meiguang Jin; Junfeng Ma

cs.SDSep 30, 2026

SCIC: Scope- and Codebook-Aware Instruction Conditioning for Speaker-Adapted Expressive TTS

Authors: Longyu Lu, Zongwei Du, Mengtao Xing, Zhuoqun Liu, Zifan Guan, Meiguang Jin, Junfeng Ma

Organizations: TaoLive-AIGC Team Taobao & Tmall Group of Alibaba

Abstract

Long-form live-streaming TTS requires context-dependent prosody and paragraph-level coherence. However, many existing instruction-based TTS systems use global or uniform conditions, providing limited explicit control over clause-level relative prosodic changes. We introduce Speaker-Relative Inline Prosody Control, where each Pitch, Energy, or Speed instruction targets a clause relative to the preceding clause from the same speaker, while Pause uses an absolute duration interval. In codec-based TTS, Speed and Pause affect sequence length, whereas Pitch and Energy rely on residual codebooks. By analyzing Qwen3-TTS RVQ codebooks, we find that Energy concentrates in early residual codebooks, whereas Pitch accumulates across a deeper prefix. We therefore propose Scope- and Codebook-Aware Instruction Conditioning (SCIC), combining a Temporal Instruction Router for frame-level tag activation with Tag-Specific Codebook Weighting over residual codebooks. SCIC improves speaker-relative Pitch and Energy control over standard instruction fine-tuning using text-token tags. We further apply multi-reward GDPO post-training to jointly optimize control and quality, improving control accuracy while preserving CER and speaker similarity. In long-form synthesis, SCIC produces a more distinct paragraph-level expressive hierarchy than speaker-adapted SFT without instructions. Audio demos are available at: https://taoliveaigc.github.io/SCIC/

Figures & tables

Figure 1: SCIC in Qwen3-TTS. The Main Talker predicts $q_{0}$ and provides $h_{t}$ . For each Pitch/Energy tag, its tag embedding is scaled by the Temporal Instruction Router output and the tag-specific codebook weight to form $s_{t,k}$ , which conditions MTP prediction of $q_{1}$ – $q_{15}$ .

Figure 2: Median speaker-relative change magnitudes after direction normalization. Down values are sign-flipped so positive values indicate the requested direction; black lines and diamonds denote frozen training-reference magnitudes.

Single-codebook transfer

Cumulative transfer

Codebook Pitch Fwd. Pitch Rev. Energy Fwd. Energy Rev.

q_{0}

0.0 [-0.0,0.1] 0.5 [-0.2,1.7] -0.0 [-0.1,0.0] -0.0 [-0.1,0.0]

q_{1}

-5.0 [-7.4,-2.8] 18.3 [15.7,21.0] 83.5 [82.7,84.3] 85.3 [84.5,86.1]

q_{2}

-0.1 [-1.6,1.2] 12.7 [10.0,16.1] 6.6 [6.1,7.0] 8.1 [7.7,8.6]

q_{3}

3.7 [2.6,4.7] 10.2 [7.7,12.9] 2.5 [2.2,2.8] 3.1 [2.7,3.5]

q_{4}

2.1 [1.4,2.9] 10.2 [7.3,13.0] 1.6 [1.3,2.0] 1.9 [1.6,2.1]

q_{5}

0.8 [0.2,1.2] 2.9 [-0.9,5.5] 1.0 [0.8,1.2] 1.2 [0.9,1.5]

Attribute Codebooks Forward Reverse Energy

q_{1}

–

q_{2}

91.0 92.1 Energy

q_{1}

–

q_{3}

94.0 94.7 Pitch

q_{1}

–

q_{4}

52.9 68.1 Pitch

q_{1}

–

q_{6}

83.4 88.7 Pitch

q_{1}

–

q_{8}

91.0 96.5 Pitch

q_{1}

–

q_{10}

95.4 99.6

Table 1: RVQ transfer (%). Left: selected single-codebook results with speaker-bootstrap 95% confidence intervals. Right: cumulative transfer over residual-codebook prefixes.

Variant	Pitch	Energy	MAE $\downarrow$
SCIC-SFT	90.14	96.39	295 ms
w/ Hard Router	88.41	96.44	375 ms
w/o Tag-Specific Codebook Weighting	89.43	94.90	292 ms

Table 3: Ablations: control-direction accuracy (%) and Router Boundary MAE.

Explore similar work

InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech
Sep 28, 2026Sihang Nie, Xueru Li, Xiaofen Xing +4Instruction Natural Language Processing
South China University of Technology · Huya Inc. · The Hongkong Polytechnic University
AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech
May 14, 2026Bin Kang, Shaoguo Wen, Yang Fan +6Text-To-Speech Synthesis Acoustic Latent Space
University of Chinese Academy of Sciences · Shenzhen Loop Area Institute · Tencent Turinglab +2
Taming Long-form Text-to-Speech
Sep 15, 2026Rongxiang Wang, Berkin Durmus, Aysegul Orhon +2Autoregressive Text-To-Speech Voice Conversion
Argmax, Inc. · University of Virginia · Bilkent University

Sep 28, 2026, eess.ASJ/K move · Enter open · S save

InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech

Sihang Nie, Xueru Li, Xiaofen Xing, Deyi Tuo+3

South China University of Technology · Huya Inc. · The Hongkong Polytechnic University

Instruction Natural Language Processing

Open paper AskPDF arXiv

Instruction-based text-to-speech (ITTS) systems enable natural-language control of expressive speech generation, but often offer limited transparency and fine-grained control over individual text units. Character-level controllable TTS systems provide explicit acoustic control, yet typically rely on user-specified acoustic attributes. To bridge this gap, we propose InstCharVoice, a unified framework that grounds natural-language instructions in character-level acoustic control. We first construct grounded instruction annotations on the WordVoice-5A-zh corpus using Qwen3-Omni. With this supervision, we train an autoregressive model to identify instruction-relevant characters and predict their acoustic attributes before generating the corresponding speech tokens. Keyword prediction and grounding-aware loss weighting help the model focus on instruction-relevant characters and attributes. Experiments show improved instruction following and keyword-level acoustic control over representative ITTS systems, with competitive speech naturalness and explicit character-level controllability. Audio samples are available at https://xxh333.github.io/instcharvoice-demo/.

Abstract

Explore similar work

InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech

AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech

Taming Long-form Text-to-Speech