We introduce TiCo, a time-controllable spoken dialogue model (SDM) that follows time-constrained instructions (e.g., "Please generate a response lasting about 15 seconds") and generates spoken responses with controllable duration. This capability is valuable for real-world spoken language systems such as voice assistants and interactive agents, where controlling response duration can improve interaction quality. However, despite their strong ability to generate natural spoken responses, existing models lack time awareness and struggle to follow duration-related instructions. To systematically evaluate this, we introduce TiCo-Bench, the first benchmark for time-controllable instruction following in SDMs, on which existing open-source and commercial models frequently fail to satisfy explicit time constraints. TiCo addresses this limitation by enabling an SDM to estimate elapsed speaking time during generation through Spoken Time Markers (STM) (e.g., <10.6 seconds>). These markers help the model maintain awareness of time and adjust the remaining content to meet the target duration. TiCo is post-trained efficiently without question-answer paired data, relying on self-generation and reinforcement learning with verifiable reward. Experimental results show that TiCo reduces duration error by 2.7x over its backbone and 1.6x over the strongest baseline, while preserving response quality.
Figures & tables
Figure 1: Overview of TiCo , a two-stage framework for time-controllable speech generation. Stage 1 (top): The model leverages self-generation to produce responses annotated with Spoken Time Markers (STMs) , which serve as supervision for learning time awareness , i.e., associating intermediate generation states with temporal progress and estimating elapsed speaking time. Stage 2 (bottom): The model is further optimized via RLVR, where rewards are derived from STMs, enabling the model to regulate response duration in real time.
Figure 2: Composition of TiCo-Bench . (a) The benchmark contains 1,000 base speech queries spanning four task categories: Question Answering, Reasoning, Creative, and Summarization. (b) Most queries fall within a short duration range, while only Summarization extends to substantially longer durations.
Table 1: TiCo-Bench evaluation on speech-query tasks under Short (10–30 s) and Long (30–60 s) settings, broken down by task category (QA, REA, CRE, SUM). Per-task and overall scores are MAPE (%); rightmost columns report GPT-score and UTMOS averaged over all subsets. Model categories: Cascaded, Commercial, Open-sourced, Proposed. Bold marks the best result per column. ∗ Kimi-Audio SUM clips exceeding the model’s 120 s context are excluded. Per-task MAE, GPT-score, and UTMOS performance are in Appendix G .
Figure 3: Duration MAE and MAPE of Qwen2.5-Omni-7B and TiCo across instructed-duration bins on two speech benchmarks, InstructS2S and UROBench. TiCo maintains consistently low error across the full duration range, while the backbone’s error grows for targets beyond 45 seconds.
Figure 4: Duration error of TiCo across instructed-duration bins, comparing two reference signals: the instructed duration tinst and the final Spoken Time Marker tlast . The close alignment indicates that the final time marker accurately estimates realized speech duration.
Variant
Rmain(g)
Rpres(g)
Rmono(g)
Rcopy(g)
Rrep(g)
MAE (s)
MAPE (%)
Main only
✓
8.39
32.21
+ Presence
✓
✓
10.18
38.89
+ Presence + Mono.
✓
✓
✓
7.66
28.10
+ Mono. + Copy
✓
✓
✓
5.30
19.07
+ Mono. + Copy + Rep.
✓
✓
✓
✓
4.71
17.16
Full
✓
✓
✓
✓
✓
4.55
15.38
Table 2: Ablation of reward components in second-stage training on a TiCo-Bench subset.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Distribution of Spoken Time Markers in the First stage training data.
Table 3: Text-query evaluation on LIFEBench and UROBench-text under Short (10–30 s) and Long (30–60 s) settings. Results are reported as MAE (seconds) / MAPE (%), with the rightmost column showing GPT-score averaged across both benchmarks. Lower is better for MAE/MAPE, and higher is better for GPT-score. Model categories are indicated by color: Cascaded, Commercial, Open-sourced, and Proposed. Bold marks the best result per column.
Figure 6: Text benchmarks: duration error of Qwen2.5-Omni-7B vs. TiCo measured against instructed duration. From left to right: LIFEBench MAE (s), LIFEBench MAPE (%), UROBench MAE (s), UROBench MAPE (%). Shaded regions indicate ± 1 SEM.
Figure 7: Text benchmarks (TiCo): duration error measured against instructed duration vs. last time marker. From left to right: LIFEBench MAE (s), LIFEBench MAPE (%), UROBench MAE (s), UROBench MAPE (%). Shaded regions indicate ± 1 SEM.
Table 4: Per-task MAE (seconds) on TiCo-Bench. Lower is better. Model categories indicated by color: Cascaded, Commercial, Open-sourced, and Proposed. Bold marks the best result in each column. ∗ For Kimi Audio on the SUM subset, audio clips longer than 120 s exceed the model’s effective context window and are excluded from evaluation.
Table 5: Per-task GPT-score (1–5 scale) on TiCo-Bench. Higher is better. Model categories indicated by color: Cascaded, Commercial, Open-sourced, and Proposed. Bold marks the best result in each column. ∗ For Kimi Audio on the SUM subset, audio clips longer than 120 s exceed the model’s effective context window and are excluded from evaluation.
Table 6: Per-task UTMOS (1–5 scale) on TiCo-Bench. Higher is better. The first row reports the ground-truth query speech as a reference and is excluded from the bold comparison. Model categories indicated by color: Cascaded, Commercial, Open-sourced, and Proposed. Bold marks the best result in each column. ∗ For Kimi Audio on the SUM subset, audio clips longer than 120 s exceed the model’s effective context window and are excluded from evaluation.
Task
Short (10s–30s)
Long (30s–60s)
QA
1.90
3.09
Reasoning
2.08
3.32
Creative
2.05
3.60
Summarization
7.45
3.62
Overall
2.65
3.21
Appendix
Table 7: Local alignment quality of Spoken Time Markers in TiCo. Each cell reports the mean absolute error in seconds between the predicted Spoken Time Marker and the corresponding ASR-aligned word timestamp.
Figure 8: Illustration of different generation patterns in spoken dialogue models (SDMs): (a) Sequential, (b) Interleaved, and (c) Parallel.
Model
Date
IR
Speech Rrep.
Pattern
Notes
dGSLM [ 66 ]
2022-03
-
Phonetic token
Direct
“Dual-tower” architecture for dual channel full-duplex modeling. Direct modeling of two-channel phonetic tokens.
SpeechGPT [ 67 ]
2023-05
Text (+R.)
Phonetic token
Sequential
Using “Chain-of-Modality”. Expands LLM vocabulary with phonetic tokens.
SPIRIT-LM [ 68 ]
2024-02
Text (+S.)
Phonetic token
Interleaved
Interleaves text and speech in one stream; expressive version adds pitch and style tokens.
Moshi [ 65 ]
2024-07
Text
Acoustic token
Parallel
Dual-channel full-duplex model. Parallel decoding text and acoustic tokens with delay pattern.
LLaMA-Omni [ 49 ]
2024-09
Text
Phonetic token
Parallel
CTC speech decoder maps LLM response states to phonetic tokens for streaming speech synthesis.
SyncLLM [ 69 ]
2024-09
-
Phonetic token
Direct
Interleaving user speech and model speech for full-duplex dialogue.
Appendix
Table 8: Spoken dialogue models (SDMs) with speech input and speech output, ordered by their first public release time. Date : First released date. IR : Intermediate representation used in the SDM. Speech Rrep. : Speech representation (prediction target) used by the model. Pattern : The pattern of how the intermediate and speech representations are generated.
Spoken Dialogue Models (SDMs) typically share a speech-token space for generation and synthesis, leading to long autoregressive sequences and costly inference. Lower-rate speech tokenizers shorten sequences but may introduce compression-quality trade-offs and costly replacement without breaking this coupling. To address this limitation, we propose CoSTA, a compatibility-preserving post-hoc framework that decouples generation and synthesis token spaces while retaining the original speech tokenizer and decoder. CoSTA extends the generation token space with compressed representations via mean-initialized vocabulary extension, pronunciation adaptation, and dialogue adaptation, followed by deterministic expansion back to the original speech-token space before synthesis. On bilingual speech-dialogue benchmarks, CoSTA achieves approximately 1.7× expanded speech-token generation speed, lower full-response latency, and higher throughput while preserving response quality. Analyses show more sustained interaction under limited budgets and a practical trade-off among compression, decoding efficiency, and quality. Further experiments demonstrate cross-architecture transfer and complementarity to low-rate tokenization.
Yingzhuo Deng, Chong Li, Bingxin Liu +6
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Wuhan AI Research
Multi-speaker dialogue TTS requires natural speech generation, consistent speaker identity, coherent cross-turn transitions, and fine-grained control of expressive attributes such as emotion, speaking rate, and loudness. These requirements are difficult to satisfy reliably with one-shot generation, especially in long-form dialogue. We propose a controllable multi-speaker dialogue TTS framework that formulates synthesis as critique-driven iterative refinement. Its speech backbone, ControlEdit-TTS, unifies instruction-following synthesis and natural-language-guided attribute editing, enabling correction of expressive errors without full regeneration. The framework further performs hierarchical utterance-level and scene-level critique, routing detected issues to editing, resynthesis, or timing adjustment. Experiments on a bilingual Chinese--English dialogue benchmark show improved utterance-level instruction following, better dialogue-level preference than direct dialogue models and agentic baselines, and more effective refinement than regeneration-only alternatives while preserving speaker identity. Ablations further confirm the benefits of scene-level critique and edit-based correction.
Latency-sensitive spoken agents must know not only which tool to call, but also when they have heard enough to act. Waiting for the turn to complete delays execution even when the request is already actionable. However, neither existing spoken agents nor spoken dialogue corpora define the earliest point at which a request becomes actionable. We introduce a synthetic spoken-dialogue pipeline that provides earliest-actionable-point supervision, marking when the observed speech grounds a tool and its required arguments. Through controlled task generation, speech synthesis and alignment, and gated annotation, we construct 26,694 dialogues totaling 786 hours across 48 domains, with disjoint training and test argument values. We additionally create an audio rendering of τ2-bench for external evaluation. We train an omni-modal language model on this corpus, enabling causal, chunk-level decisions and partial-transcript generation for downstream call completion. The model achieves tool-state macro-F1 scores of 0.48 and 0.47 on held-out dialogues and the external benchmark, respectively. For correct calls, the estimated mean lead over a turn-end cascade, using the same formatter, is 1.8-2.0 seconds in-domain and 1.1 seconds on τ2-bench. These results demonstrate a data-driven path toward spoken agents that act while listening. We will open-source the pipeline and the training scripts.
Yizhou Peng, Chen Chen, Donghang Wu +2
Nanyang Technological University, Singapore · AI Singapore, Singapore