TiCo: Time-Controllable Spoken Dialogue Model

Kai-Wei Chang; Wei-Chih Chen; En-Pei Hu; Hung-yi Lee; James Glass

cs.CLMar 23, 2026

TiCo: Time-Controllable Spoken Dialogue Model

Authors: Kai-Wei Chang, Wei-Chih Chen, En-Pei Hu, Hung-yi Lee, James Glass

Organizations: MIT · NTU · NTU AI-CoRE

Abstract

We introduce TiCo, a time-controllable spoken dialogue model (SDM) that follows time-constrained instructions (e.g., "Please generate a response lasting about 15 seconds") and generates spoken responses with controllable duration. This capability is valuable for real-world spoken language systems such as voice assistants and interactive agents, where controlling response duration can improve interaction quality. However, despite their strong ability to generate natural spoken responses, existing models lack time awareness and struggle to follow duration-related instructions. To systematically evaluate this, we introduce TiCo-Bench, the first benchmark for time-controllable instruction following in SDMs, on which existing open-source and commercial models frequently fail to satisfy explicit time constraints. TiCo addresses this limitation by enabling an SDM to estimate elapsed speaking time during generation through Spoken Time Markers (STM) (e.g., <10.6 seconds>). These markers help the model maintain awareness of time and adjust the remaining content to meet the target duration. TiCo is post-trained efficiently without question-answer paired data, relying on self-generation and reinforcement learning with verifiable reward. Experimental results show that TiCo reduces duration error by 2.7x over its backbone and 1.6x over the strongest baseline, while preserving response quality.

Figures & tables

Figure 1: Overview of TiCo , a two-stage framework for time-controllable speech generation. Stage 1 (top): The model leverages self-generation to produce responses annotated with Spoken Time Markers (STMs) , which serve as supervision for learning time awareness , i.e., associating intermediate generation states with temporal progress and estimating elapsed speaking time. Stage 2 (bottom): The model is further optimized via RLVR, where rewards are derived from STMs, enabling the model to regulate response duration in real time.

Panel of Figure 2 — Figure 2: Composition of TiCo-Bench . (a) The benchmark contains 1,000 base speech queries spanning four task categories: Question Answering, Reasoning, Creative, and Summarization. (b) Most queries fall within a short duration range, while only Summarization extends to substantially longer durations.

Panel of Table 1 — Table 1: TiCo-Bench evaluation on speech-query tasks under Short (10–30 s) and Long (30–60 s) settings, broken down by task category (QA, REA, CRE, SUM). Per-task and overall scores are MAPE (%); rightmost columns report GPT-score and UTMOS averaged over all subsets. Model categories: Cascaded, Commercial, Open-sourced, Proposed. Bold marks the best result per column. ∗ Kimi-Audio SUM clips exceeding the model’s 120 s context are excluded. Per-task MAE, GPT-score, and UTMOS performance are in Appendix G .

Figure 3: Duration MAE and MAPE of Qwen2.5-Omni-7B and TiCo across instructed-duration bins on two speech benchmarks, InstructS2S and UROBench. TiCo maintains consistently low error across the full duration range, while the backbone’s error grows for targets beyond 45 seconds.

Figure 4: Duration error of TiCo across instructed-duration bins, comparing two reference signals: the instructed duration $t_{\mathrm{inst}}$ and the final Spoken Time Marker $t_{\mathrm{last}}$ . The close alignment indicates that the final time marker accurately estimates realized speech duration.

Variant	$\mathcal{R}^{(g)}_{\text{main}}$	$\mathcal{R}^{(g)}_{\text{pres}}$	$\mathcal{R}^{(g)}_{\text{mono}}$	$\mathcal{R}^{(g)}_{\text{copy}}$	$\mathcal{R}^{(g)}_{\text{rep}}$	MAE (s)	MAPE (%)
Main only	✓					8.39	32.21
+ Presence	✓	✓				10.18	38.89
+ Presence + Mono.	✓	✓	✓			7.66	28.10
+ Mono. + Copy	✓		✓	✓		5.30	19.07
+ Mono. + Copy + Rep.	✓		✓	✓	✓	4.71	17.16
Full	✓	✓	✓	✓	✓	4.55	15.38

Table 2: Ablation of reward components in second-stage training on a TiCo-Bench subset.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Distribution of Spoken Time Markers in the First stage training data.

Panel of Table 3 — Table 3: Text-query evaluation on LIFEBench and UROBench-text under Short (10–30 s) and Long (30–60 s) settings. Results are reported as MAE (seconds) / MAPE (%), with the rightmost column showing GPT-score averaged across both benchmarks. Lower is better for MAE/MAPE, and higher is better for GPT-score. Model categories are indicated by color: Cascaded, Commercial, Open-sourced, and Proposed. Bold marks the best result per column.

Figure 6: Text benchmarks: duration error of Qwen2.5-Omni-7B vs. TiCo measured against instructed duration. From left to right: LIFEBench MAE (s), LIFEBench MAPE (%), UROBench MAE (s), UROBench MAPE (%). Shaded regions indicate $\pm$ 1 SEM.

Figure 7: Text benchmarks (TiCo): duration error measured against instructed duration vs. last time marker. From left to right: LIFEBench MAE (s), LIFEBench MAPE (%), UROBench MAE (s), UROBench MAPE (%). Shaded regions indicate $\pm$ 1 SEM.

Panel of Table 4 — Table 4: Per-task MAE (seconds) on TiCo-Bench. Lower is better. Model categories indicated by color: Cascaded, Commercial, Open-sourced, and Proposed. Bold marks the best result in each column. ∗ For Kimi Audio on the SUM subset, audio clips longer than 120 s exceed the model’s effective context window and are excluded from evaluation.

Panel of Table 5 — Table 5: Per-task GPT-score (1–5 scale) on TiCo-Bench. Higher is better. Model categories indicated by color: Cascaded, Commercial, Open-sourced, and Proposed. Bold marks the best result in each column. ∗ For Kimi Audio on the SUM subset, audio clips longer than 120 s exceed the model’s effective context window and are excluded from evaluation.

Panel of Table 6 — Table 6: Per-task UTMOS (1–5 scale) on TiCo-Bench. Higher is better. The first row reports the ground-truth query speech as a reference and is excluded from the bold comparison. Model categories indicated by color: Cascaded, Commercial, Open-sourced, and Proposed. Bold marks the best result in each column. ∗ For Kimi Audio on the SUM subset, audio clips longer than 120 s exceed the model’s effective context window and are excluded from evaluation.

Task	Short (10s–30s)	Long (30s–60s)
QA	1.90	3.09
Reasoning	2.08	3.32
Creative	2.05	3.60
Summarization	7.45	3.62
Overall	2.65	3.21

Appendix

Table 7: Local alignment quality of Spoken Time Markers in TiCo. Each cell reports the mean absolute error in seconds between the predicted Spoken Time Marker and the corresponding ASR-aligned word timestamp.

Figure 8: Illustration of different generation patterns in spoken dialogue models (SDMs): (a) Sequential, (b) Interleaved, and (c) Parallel.

Model	Date	IR	Speech Rrep.	Pattern	Notes
dGSLM [ 66 ]	2022-03	-	Phonetic token	Direct	“Dual-tower” architecture for dual channel full-duplex modeling. Direct modeling of two-channel phonetic tokens.
SpeechGPT [ 67 ]	2023-05	Text (+R.)	Phonetic token	Sequential	Using “Chain-of-Modality”. Expands LLM vocabulary with phonetic tokens.
SPIRIT-LM [ 68 ]	2024-02	Text (+S.)	Phonetic token	Interleaved	Interleaves text and speech in one stream; expressive version adds pitch and style tokens.
Moshi [ 65 ]	2024-07	Text	Acoustic token	Parallel	Dual-channel full-duplex model. Parallel decoding text and acoustic tokens with delay pattern.
LLaMA-Omni [ 49 ]	2024-09	Text	Phonetic token	Parallel	CTC speech decoder maps LLM response states to phonetic tokens for streaming speech synthesis.
SyncLLM [ 69 ]	2024-09	-	Phonetic token	Direct	Interleaving user speech and model speech for full-duplex dialogue.

Appendix

Table 8: Spoken dialogue models (SDMs) with speech input and speech output, ordered by their first public release time. Date : First released date. IR : Intermediate representation used in the SDM. Speech Rrep. : Speech representation (prediction target) used by the model. Pattern : The pattern of how the intermediate and speech representations are generated.

Spoken Dialogue Systems

Spoken dialogue systems let machines understand and respond to speech while managing turn-taking, context, emotion, and real-time interaction.

See all 141 papers

CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models
Oct 15, 2026Yingzhuo Deng, Chong Li, Bingxin Liu +6Spoken Dialogue Systems Speech Language Models
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Wuhan AI Research
From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS
Sep 27, 2026Kangxiang Xia, Xinfa Zhu, HangRui Hu +8TTS Synthesis Speech Editing
Independent Researcher
Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems
Oct 26, 2026Yizhou Peng, Chen Chen, Donghang Wu +2Spoken Dialogue Systems Spoken Language Understanding
Nanyang Technological University, Singapore · AI Singapore, Singapore

Oct 15, 2026J/K move · Enter open · S save

CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models

Yingzhuo Deng, Chong Li, Bingxin Liu, Yi Shu+5

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Wuhan AI Research

Spoken Dialogue Systems Speech Language Models Audio Token Compression

Open paper AskPDF arXiv

Spoken Dialogue Models (SDMs) typically share a speech-token space for generation and synthesis, leading to long autoregressive sequences and costly inference. Lower-rate speech tokenizers shorten sequences but may introduce compression-quality trade-offs and costly replacement without breaking this coupling. To address this limitation, we propose CoSTA, a compatibility-preserving post-hoc framework that decouples generation and synthesis token spaces while retaining the original speech tokenizer and decoder. CoSTA extends the generation token space with compressed representations via mean-initialized vocabulary extension, pronunciation adaptation, and dialogue adaptation, followed by deterministic expansion back to the original speech-token space before synthesis. On bilingual speech-dialogue benchmarks, CoSTA achieves approximately

1.7\times

expanded speech-token generation speed, lower full-response latency, and higher throughput while preserving response quality. Analyses show more sustained interaction under limited budgets and a practical trade-off among compression, decoding efficiency, and quality. Further experiments demonstrate cross-architecture transfer and complementarity to low-rate tokenization.

Abstract

Spoken Dialogue Systems

Explore similar work

CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models

From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS

Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems