CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models

Yingzhuo Deng; Chong Li; Bingxin Liu; Yi Shu; Tianyu Peng; Can Xie; Siyao Song; Shuo Ren

cs.CLOct 15, 2026

CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models

Authors: Yingzhuo Deng, Chong Li, Bingxin Liu, Yi Shu, Tianyu Peng, Can Xie, Siyao Song, Shuo Ren,

Organizations: Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Wuhan AI Research

Abstract

Spoken Dialogue Models (SDMs) typically share a speech-token space for generation and synthesis, leading to long autoregressive sequences and costly inference. Lower-rate speech tokenizers shorten sequences but may introduce compression-quality trade-offs and costly replacement without breaking this coupling. To address this limitation, we propose CoSTA, a compatibility-preserving post-hoc framework that decouples generation and synthesis token spaces while retaining the original speech tokenizer and decoder. CoSTA extends the generation token space with compressed representations via mean-initialized vocabulary extension, pronunciation adaptation, and dialogue adaptation, followed by deterministic expansion back to the original speech-token space before synthesis. On bilingual speech-dialogue benchmarks, CoSTA achieves approximately $1.7\times$ expanded speech-token generation speed, lower full-response latency, and higher throughput while preserving response quality. Analyses show more sustained interaction under limited budgets and a practical trade-off among compression, decoding efficiency, and quality. Further experiments demonstrate cross-architecture transfer and complementarity to low-rate tokenization.

Figures & tables

Figure 1: Overview of CoSTA. Existing SDMs couple generation and synthesis token spaces. CoSTA extends the generation token space with compressed representations and deterministically expands them back to the original space before synthesis.

Figure 2: Speech-token BPE scaling across eight tokenizer configurations. Compression rate decreases approximately linearly with

\log_{2}

vocabulary size.

Figure 3: CoSTA combines vocabulary extension with mean-based initialization, progressive pronunciation adaptation, and dialogue adaptation.

Figure 4: Inference with deterministic expansion from the extended generation token space to the original space before unchanged decoding.

Model	AlpacaEval-zh		APE-zh		Claude-zh		LCSTS-zh		MLC-zh		Wildchat-zh		Avg
Model	$\boldsymbol{\rho_{g}}\downarrow$	Speed $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	Speed $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	Speed $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	Speed $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	Speed $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	Speed $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	Speed $\uparrow$	Relative $\uparrow$
Step-Audio 2 mini	100.00%	55.51	100.00%	60.88	100.00%	56.39	100.00%	57.05	100.00%	65.22	100.00%	57.40	100.00%	58.74	100.00%
Vanilla-FT-ZH	100.00%	57.84	100.00%	60.40	100.00%	54.99	100.00%	55.54	100.00%	63.47	100.00%	59.10	100.00%	58.56	99.69%
Vanilla-FT-MIX	100.00%	59.01	100.00%	59.65	100.00%	56.93	100.00%	56.03	100.00%	61.57	100.00%	59.08	100.00%	58.71	99.95%
CoSTA-ZH	67.50%	103.03	64.63%	101.68	68.52%	101.20	69.40%	94.46	62.70%	110.31	68.13%	101.29	66.81%	101.99	173.63%
CoSTA-MIX	69.12%	98.44	64.68%	102.34	70.58%	92.54	71.06%	91.37	63.03%	106.32	70.21%	93.22	68.11%	97.37	165.76%

Table 1: Model-side efficiency on Chinese and English speech-to-speech benchmarks.

\rho_{g}

denotes generated-token compression, Speed denotes expanded speech-token generation speed, and Relative normalizes Speed to Step-Audio 2 mini within each language setting.

Concurrency	Model	Full-Response Latency (s) $\downarrow$	Throughput (req/s) $\uparrow$	Peak Memory (GiB)
1	Step-Audio 2 mini / Vanilla-FT-ZH / CoSTA-ZH	10.73 / 9.72 / 7.14	0.103 / 0.114 / 0.160	78.12 / 77.64 / 78.56
4	Step-Audio 2 mini / Vanilla-FT-ZH / CoSTA-ZH	13.63 / 11.38 / 9.16	0.316 / 0.369 / 0.479	78.34 / 78.13 / 79.02
8	Step-Audio 2 mini / Vanilla-FT-ZH / CoSTA-ZH	15.56 / 13.71 / 9.49	0.515 / 0.601 / 0.820	78.90 / 79.17 / 79.14
1	Step-Audio 2 mini / Vanilla-FT-EN / CoSTA-EN	12.50 / 16.16 / 9.25	0.088 / 0.068 / 0.122	78.98 / 77.16 / 78.69
4	Step-Audio 2 mini / Vanilla-FT-EN / CoSTA-EN	15.52 / 20.17 / 11.59	0.276 / 0.212 / 0.372	79.24 / 78.33 / 79.12
8	Step-Audio 2 mini / Vanilla-FT-EN / CoSTA-EN	18.62 / 25.04 / 14.44	0.442 / 0.330 / 0.556	78.52 / 77.73 / 79.04

Table 2: System-level efficiency on AlpacaEval-zh (upper) and AlpacaEval (lower) at concurrency 1, 4, and 8 on a single A100 80GB GPU. Values are ordered as Step-Audio 2 mini, Vanilla-FT, and CoSTA.

Model	AlpacaEval-zh	APE-zh	Claude-zh	LCSTS-zh	MLC-zh	Wildchat-zh	Avg
Model	AlpacaEval-zh	APE-zh	Claude-zh	LCSTS-zh	MLC-zh	Wildchat-zh	G-Eval $\uparrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$
Qwen2.5-Omni	74.10	62.11	78.86	69.75	82.53	78.35	74.28	2.93%	2.95
Kimi-Audio	77.05	39.12	80.24	76.13	80.92	78.08	71.93	4.88%	2.52
GLM-4-Voice	84.76	23.16	83.54	76.36	78.85	84.41	71.85	3.77%	2.94
Step-Audio 2 mini	80.18	21.93	72.40	79.16	78.16	83.59	69.24	7.49%	3.62
Vanilla-FT-ZH	79.77	21.40	80.35	78.82	77.47	81.47	69.88	4.96%	3.77

Table 3: Automatic quality evaluation on Chinese and English speech-to-speech benchmarks. Benchmark columns report G-Eval; Avg reports average G-Eval, ASR-CER/WER, and UTMOS. Bold and underline mark the best and second-best results among Step-Audio 2 mini variants; external models serve as quality references.

Figure 5: Paired human evaluation of CoSTA against the base model and Vanilla-FT. Bars show wins, ties, and losses from 180 paired MOS ratings per language and metric.

Variant	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$
Step-Audio 2 mini	100.00%	7.49%	3.62	69.24
Vanilla-FT-ZH	100.00%	4.96%	3.77	69.88
CoSTA-ZH	66.73%	1.51%	3.75	72.27
w/o Timbre Conversion	67.01%	1.52%	3.75	70.93
w/o Progressive Adaptation	67.75%	1.46%	3.75	72.19
w/o Pronunciation Adaptation	69.56%	1.74%	3.74	72.12

Table 4: Component ablation on Chinese benchmarks.

\rho_{g}

, ASR-CER, UTMOS, and G-Eval measure generated-token compression, text-speech alignment, speech quality, and response-content quality, respectively.

Figure 6: Training loss during the first stage of pronunciation adaptation. Mean initialization achieves faster early convergence than the alternatives.

Figure 7: Vocabulary-size trade-off on AlpacaEval-zh: relative expanded speech-token generation speed with (a) UTMOS and (b) G-Eval. The 0K baseline denotes the original Step-Audio 2 mini; other points denote CoSTA-ZH.

Model	# Turns $\uparrow$	Gen Tokens	Final Tokens	Duration (s)	$\boldsymbol{\rho_{g}}\downarrow$
Step-Audio 2 mini	2.14	2208.21	2208.21	88.33	100.00%
Vanilla-FT-ZH	3.71	2440.50	2440.50	97.62	100.00%
CoSTA-ZH	7.21	2500.79	3766.86	150.67	66.39%
Step-Audio 2 mini	2.58	2218.74	2218.74	88.75	100.00%
Vanilla-FT-EN	1.84	1894.79	1894.79	75.79	100.00%
CoSTA-EN	5.79	2608.53	3650.79	146.03	71.45%

Table 5: Interaction sustainability under a limited generation budget. Gen/Final Tokens denote speech tokens before/after deterministic expansion; Duration denotes generated speech duration.

Model	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER/WER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$
OpenS2S	100.00%	11.08%	2.41	65.26
Vanilla-FT-ZH	100.00%	11.36%	2.38	67.39
CoSTA-ZH	78.83%	8.50%	2.36	69.18
OpenS2S	100.00%	4.65%	3.81	73.97
Vanilla-FT-EN	100.00%	5.16%	3.79	75.48
CoSTA-EN	81.36%	4.66%	3.78	76.23

Table 6: Transfer of CoSTA to OpenS2S. Upper/lower blocks report Chinese/English results using ASR-CER/WER, respectively. Matched Vanilla-FT uses the same architecture and language-specific adaptation data.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Data split	50% / 50%
Stage-1 trainable params	Embedding, LM head
Stage-2 trainable params	+ Intermediate layers
Optimizer	AdamW
Stage-1 learning rate	$1.6\times 10^{-4}$
Stage-2 learning rate	$2.0\times 10^{-5}$

Appendix

Table 7: Hyperparameters used during pronunciation adaptation.

Hyperparameter	Value
Trainable params	All LLM parameters
Optimizer	AdamW
Learning rate	$2.0\times 10^{-5}$
Scheduler	Cosine
Warmup ratio	0.03
Global batch size	256

Appendix

Table 8: Hyperparameters used during dialogue adaptation.

Hyperparameter	Value
Decoding mode	Greedy
Repetition penalty	1.2
Max new tokens	2048
BPE expansion	Enabled
Expansion target	Speech-token stream only
Text-token stream	Unchanged

Appendix

Table 9: Hyperparameters used during inference.

Data Type	Language	Scale
Text-to-speech	Chinese	300h
Text-to-speech	English	300h
Speech dialogue	Chinese	49,664 examples
Speech dialogue	English	49,408 examples

Appendix

Table 10: Statistics of training data.

Dataset	#Samples	Task / Aspect	Eval. Mode
AlpacaEval-zh	147	Authentic, open-ended dialogue	Open
AlpacaEval	199	Authentic, open-ended dialogue	Open
APE-zh	190	Practical mathematical problems	QA
Gsm8kEval	582	Practical mathematical problems	QA
Claude-zh	222	Authentic, open-ended dialogue	Open
CommonEval	200	Authentic, open-ended dialogue	Open

Appendix

Table 11: URO-Bench datasets used in our evaluation. The selected datasets cover understanding, reasoning, and oral conversation in Chinese and English speech-to-speech settings.

Figure 8: Gradient norm during the first stage of pronunciation adaptation. Mean initialization yields more stable optimization than the alternatives.

Figure 9: Held-out compression across data sources and languages with S3Tokenizer V2. Points are test measurements; lines are fits against log added vocabulary size. Emilia-TC denotes timbre-converted Emilia.

Figure 10: Training-corpus compression across eight tokenizer configurations on Chinese dialogue speech, corresponding to the test results in Figure 2 . Points are measurements and lines are log-linear fits. Mimi and Qwen3-TTS use their first semantic codebook.

Tokenizer	Public reference source	Codes	Hz	Token stream used for BPE
GLM-4-Voice Tokenizer 12.5Hz	GLM-4-Voice tokenizer	16,384	12.5	Single quantized speech stream
S3Tokenizer V2 25Hz	CosyVoice2-0.5B	6,561	25	Single semantic-token stream
S3Tokenizer V3 25Hz	Fun-CosyVoice3-0.5B-2512	6,561	25	Single semantic-token stream
Mimi	kyutai/mimi	2,048	12.5	First semantic codebook only
S3Tokenizer V1 25Hz	CosyVoice-300M-25Hz	4,096	25	Single semantic-token stream
Qwen3-TTS-Tokenizer-12Hz	Qwen3-TTS tokenizer	2,048	12.5	First semantic codebook only

Appendix

Table 12: Tokenizer configurations for Chinese dialogue compression, using the figure names and linked public sources. Codes excludes added BPE units. Hz is the native rate; mHuBERT is omitted. Qwen3-TTS operates at 12.5 Hz despite its model name ( official configuration ).

Figure 11: Training-corpus compression across data sources and languages with S3Tokenizer V2, corresponding to the test results in Figure 9 . Points are measurements and lines are log-linear fits. Each vocabulary is learned on the training corpus represented by its curve.

Name	License / Terms
Step-Audio 2 mini	Apache 2.0 license
Qwen2.5-Omni	Apache 2.0 license
Kimi-Audio	Apache 2.0 / MIT licenses
GLM-4-Voice / Tokenizer	glm-4-voice Model License
OpenS2S	Apache 2.0 license
Emilia	CC BY-NC 4.0 license

Appendix

Table 13: Licenses and usage terms of scientific artifacts used in this work.

Model	AlpacaEval-zh			APE-zh			Claude-zh			Avg
Model	CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$
Qwen2.5-Omni	2.09%	2.93	74.10	8.50%	3.02	62.11	2.47%	2.73	78.86	2.93%	2.95	74.28
Kimi-Audio	4.71%	2.52	77.05	8.29%	2.59	39.12	11.12%	2.08	80.24	4.88%	2.52	71.93
GLM-4-Voice	3.65%	2.86	84.76	5.51%	3.14	23.16	7.13%	2.59	83.54	3.77%	2.94	71.85
Step-Audio 2 mini	8.48%	3.58	80.18	4.35%	3.87	21.93	20.28%	3.33	72.40	7.49%	3.62	69.24
Vanilla-FT-ZH	5.11%	3.76	79.77	0.95%	4.06	21.40	19.51%	3.41	80.35	4.96%	3.77	69.88

Appendix

Table 14: Detailed quality results on Chinese speech-to-speech benchmarks. Each benchmark reports CER, UTMOS, and G-Eval for text-speech alignment, speech quality, and response-content quality, respectively.

Model	AlpacaEval			Gsm8kEval			CommonEval			Avg
Model	WER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	WER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	WER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	WER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$
Qwen2.5-Omni	2.05%	4.22	76.78	4.25%	4.32	54.30	1.48%	4.27	72.00	1.73%	4.28	72.88
Kimi-Audio	20.77%	2.53	62.01	99.08%	2.77	0.57	37.62%	2.95	50.77	34.92%	2.86	50.46
GLM-4-Voice	13.92%	4.00	80.87	8.29%	4.25	29.32	26.28%	4.10	66.77	11.55%	4.16	69.95
Step-Audio 2 mini	9.82%	4.51	76.31	17.07%	4.53	15.92	1.64%	4.53	52.53	6.49%	4.53	63.73
Vanilla-FT-EN	23.82%	4.35	73.74	4.15%	4.53	16.67	9.73%	4.47	72.03	10.16%	4.46	66.59

Appendix

Table 15: Detailed quality results on English speech-to-speech benchmarks. Each benchmark reports WER, UTMOS, and G-Eval for text-speech alignment, speech quality, and response-content quality, respectively.

Model	TTS Data	AlpacaEval-zh				APE-zh				Avg
Model	TTS Data	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$
Step-Audio 2 mini	–	100.00%	8.48%	3.54	79.95	100.00%	4.28%	3.87	22.11	100.00%	8.57%	3.66	63.34
CoSTA-ZH	500h	68.17%	1.62%	3.70	80.86	65.32%	1.11%	4.06	21.05	66.48%	1.52%	3.79	66.35
	400h	68.73%	1.69%	3.70	82.40	65.88%	1.48%	4.05	25.26	67.35%	1.62%	3.79	67.71
	300h	68.54%	1.91%	3.72	81.59	65.09%	1.02%	4.06	25.09	66.83%	1.66%	3.78	67.75
	200h	69.29%	2.15%	3.68	82.36	65.69%	0.83%	4.07	26.84	67.24%	1.71%	3.78	67.99

Appendix

Table 16: Effect of pronunciation-adaptation data scale on Chinese benchmarks. Dialogue-adaptation data is fixed to 50K examples.

\rho_{g}

denotes the generated-token compression rate, while ASR-CER, UTMOS, and G-Eval evaluate text-speech alignment, speech quality, and response-content quality, respectively.

Model	Dialogue Data	AlpacaEval-zh				APE-zh				Avg
Model	Dialogue Data	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$
Step-Audio 2 mini	–	100.00%	8.48%	3.54	79.95	100.00%	4.28%	3.87	22.11	100.00%	8.57%	3.66	63.34
CoSTA-ZH	50K	68.54%	1.91%	3.72	81.59	65.09%	1.02%	4.06	25.09	66.83%	1.66%	3.78	67.75
	40K	69.24%	1.79%	3.71	78.59	65.61%	1.36%	4.04	22.81	67.28%	1.72%	3.78	66.02
	30K	68.82%	1.90%	3.73	79.23	66.04%	1.00%	4.04	18.07	67.50%	1.80%	3.78	64.95
	20K	68.32%	2.02%	3.70	80.45	66.20%	0.95%	4.06	25.44	67.49%	1.97%	3.79	66.84

Appendix

Table 17: Effect of dialogue-adaptation data scale on Chinese benchmarks. Pronunciation-adaptation data is fixed to 300h.

\rho_{g}

denotes the generated-token compression rate, while ASR-CER, UTMOS, and G-Eval evaluate text-speech alignment, speech quality, and response-content quality, respectively.

Model	$\boldsymbol{\rho_{g}}\downarrow$	Speed $\uparrow$	Prefill (ms) $\downarrow$	Decode (ms) $\downarrow$	LLM Total (ms) $\downarrow$	Speech Decoder (ms) $\downarrow$	Total (ms) $\downarrow$
Step-Audio 2 mini	100.00%	58.74	56.02	164.42	220.44	337.46	557.90
Vanilla-FT-ZH	100.00%	58.56	55.37	164.22	219.59	340.51	560.10
CoSTA-ZH	66.81%	101.99	54.55	122.98	177.53	328.06	505.59
Step-Audio 2 mini	100.00%	59.69	53.07	165.08	218.15	344.44	562.60
Vanilla-FT-EN	100.00%	57.67	55.72	165.10	220.82	339.52	560.34
CoSTA-EN	70.17%	99.05	53.24	115.96	169.20	339.17	508.36

Appendix

Table 18: First-packet latency breakdown for Chinese and English settings. The upper and lower blocks report Chinese and English results, respectively.

Metric	Model	AlpacaEval-zh	APE-zh	Claude-zh	LCSTS-zh	MLC-zh	Wildchat-zh	Avg.
Speech Fluency	Step-Audio 2 mini	4.70	4.77	3.77	4.47	4.83	4.37	4.48
	Vanilla-FT-ZH	3.80	4.50	3.60	4.83	4.60	4.77	4.35
	CoSTA-ZH	4.73	4.33	4.60	4.67	4.70	4.73	4.63
Semantic Quality	Step-Audio 2 mini	4.17	2.23	3.17	3.60	3.83	4.10	3.52
	Vanilla-FT-ZH	4.00	1.83	3.43	3.80	3.70	4.23	3.50
	CoSTA-ZH	4.40	2.17	3.60	4.10	4.00	4.23	3.75

Appendix

Table 19: MOS-style human evaluation on Chinese URO-Bench datasets. Scores are averaged over six evaluators and five randomly sampled queries per dataset. Higher scores indicate better performance.

Metric	Model	AlpacaEval	Gsm8kEval	CommonEval	Summary	MLC	WildchatEval	Avg.
Speech Fluency	Step-Audio 2 mini	4.23	4.03	4.73	4.70	4.50	4.80	4.50
	Vanilla-FT-EN	3.53	4.63	4.23	4.77	4.30	4.37	4.31
	CoSTA-EN	4.80	4.70	4.77	4.87	4.83	4.73	4.78
Semantic Quality	Step-Audio 2 mini	3.50	2.33	1.23	4.57	4.20	4.27	3.35
	Vanilla-FT-EN	3.13	1.77	4.00	4.40	4.80	4.27	3.73
	CoSTA-EN	4.37	1.20	4.17	4.90	4.83	4.73	4.03

Appendix

Table 20: MOS-style human evaluation on English URO-Bench datasets. Scores are averaged over six evaluators and five randomly sampled queries per dataset. Higher scores indicate better performance.

Model	AlpacaEval-zh				Claude-zh				LCSTS-zh				Avg
Model	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	ASR-CER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$
OpenS2S	100.00%	9.02%	2.54	69.71	100.00%	14.40%	2.18	66.97	100.00%	9.82%	2.51	59.10	100.00%	11.08%	2.41	65.26
Vanilla-FT-ZH	100.00%	6.26%	2.49	72.29	100.00%	20.00%	2.14	66.64	100.00%	7.81%	2.50	63.25	100.00%	11.36%	2.38	67.39
CoSTA-ZH	78.34%	5.98%	2.52	70.61	79.92%	11.77%	2.08	71.65	78.23%	7.75%	2.49	65.27	78.83%	8.50%	2.36	69.18
Model	AlpacaEval				CommonEval				Summary				Avg
Model	$\boldsymbol{\rho_{g}}\downarrow$	ASR-WER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	ASR-WER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	ASR-WER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$	$\boldsymbol{\rho_{g}}\downarrow$	ASR-WER $\downarrow$	UTMOS $\uparrow$	G-Eval $\uparrow$

Appendix

Table 21: Detailed SDM-level generalization results on OpenS2S. CoSTA is compared with the original OpenS2S model and matched vanilla fine-tuning on Chinese and English benchmarks.

\rho_{g}

denotes the generated-token compression rate, while ASR-CER (Chinese) / ASR-WER (English), UTMOS, and G-Eval evaluate text-speech alignment, speech quality, and response-content quality, respectively.

Model	Original	BPE	Original/BPE	BPE coverage	Avg. expansion
CoSTA-ZH	70.95%	29.05%	2.44	51.64%	2.608
w/o Dialogue Adaptation	78.95%	21.05%	3.75	36.71%	2.175

Appendix

Table 22: Speech-token usage with and without dialogue adaptation on all six Chinese benchmarks (1,122 responses per model). Statistics are pooled over generated speech tokens. (a) Original/BPE shares and their ratio use counts before expansion; BPE coverage uses counts after expansion. Avg. expansion is the number of original tokens per generated BPE unit. (b)

L

is the expansion length, with

L=1

denoting an original token. Each percentage uses all generated speech tokens before expansion as its denominator.

Model	Size	$\boldsymbol{\rho_{g}}\downarrow$	Speed $\uparrow$	Relative $\uparrow$	UTMOS $\uparrow$	G-Eval $\uparrow$
Step-Audio 2 mini	0K	100.00%	55.51	100.00%	3.58	80.18
CoSTA-ZH	10K	76.66%	78.67	141.71%	3.73	80.86
	20K	74.33%	84.51	152.24%	3.71	80.73
	40K	71.25%	93.20	167.89%	3.69	83.35
	80K	67.50%	103.03	185.59%	3.70	81.36
	160K	63.62%	104.18	187.65%	3.68	78.87

Appendix

Table 23: Complete vocabulary-size results on AlpacaEval-zh underlying Figure 7 . Size is the number of added BPE units; Speed is expanded speech tokens/s, normalized to the original Step-Audio 2 mini in Relative. The 0K row is the unadapted baseline.

Added units	Compression $\downarrow$	Median ms/step $\downarrow$	Overhead $\downarrow$	Norm. Cost $\downarrow$
0K	100.00%	0.6902	+0.00%	1.000
10K	73.27%	0.7320	+6.06%	0.777
20K	69.79%	0.7739	+12.13%	0.783
40K	66.15%	0.8576	+24.25%	0.822
80K	62.29%	1.0251	+48.52%	0.925
160K	58.48%	1.3598	+97.01%	1.152

Appendix

Table 24: Complete corpus-compression and isolated LM-head projection plus greedy-argmax costs for 0K–320K added BPE units, measured on a single A100 GPU with identical BF16 hidden states at batch size 1. Compression uses held-out Dialogue-ZH token counts. Derived costs use unrounded compression rates. Overhead is per-step latency relative to 0K; Norm. Cost is

\rho_{M}t_{M}/t_{0}

for a fixed expanded token count (Equation 9 ).

Tokenizer	Split	$\alpha$ (%)	$\beta$ (pp/doubling)	$R^{2}$	RMSE (pp)	Max $\|e\|$ (pp)
GLM-4-Voice Tokenizer 12.5Hz	Train	84.884	-4.808	0.9959	0.528	0.784
	Test	84.866	-4.193	0.9989	0.242	0.386
S3Tokenizer V2 25Hz	Train	73.550	-4.013	0.9988	0.241	0.370
	Test	73.397	-3.698	0.9997	0.103	0.149
S3Tokenizer V3 25Hz	Train	71.678	-4.107	0.9999	0.085	0.138
	Test	71.569	-3.809	0.9991	0.191	0.309

Appendix

Table 25: Complete training and test fits for the eight tokenizer configurations on Chinese dialogue speech. Full names, configurations, and public reference sources are given in Table 12 . Fits use

y=\alpha+\beta\log_{2}(M/10\mathrm{K})

, where

y=100\rho

. Each row fits six vocabulary settings; errors are in percentage points.

Setting	Split	$\alpha$ (%)	$\beta$ (pp/doubling)	$R^{2}$	RMSE (pp)	Max $\|e\|$ (pp)
Emilia-MIX	Train	85.902	-3.653	0.9929	0.529	0.752
	Test	85.664	-3.621	0.9942	0.474	0.670
Emilia-EN	Train	85.513	-3.706	0.9942	0.485	0.693
	Test	85.127	-3.680	0.9957	0.414	0.605
Emilia-ZH	Train	84.656	-3.751	0.9947	0.468	0.676
	Test	84.464	-3.681	0.9954	0.426	0.618

Appendix

Table 26: Complete training and test fits across data sources and languages with S3Tokenizer V2. Fits use

y=\alpha+\beta\log_{2}(M/10\mathrm{K})

, where

y=100\rho

. Each row fits six vocabulary settings; errors are in percentage points.

Spoken Dialogue Systems

Spoken dialogue systems let machines understand and respond to speech while managing turn-taking, context, emotion, and real-time interaction.

See all 140 papers

TiCo: Time-Controllable Spoken Dialogue Model
Mar 23, 2026Kai-Wei Chang, Wei-Chih Chen, En-Pei Hu +2Spoken Dialogue Systems Instruction Following
MIT · NTU · NTU AI-CoRE
Full-duplex Spoken Language Modeling with Discrete Diffusion
Oct 13, 2026Chen Chen, Pin-Jui Ku, Ankita Pasad +3Speech Language Models Full-Duplex Spoken Dialogue Systems
NVIDIA
FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation
Jan 8, 2026Junseok Lee, Chang-Jae ChunMultimodal Large Language Models Speech Processing
OKESTRO · Sejong University

Mar 23, 2026J/K move · Enter open · S save

TiCo: Time-Controllable Spoken Dialogue Model

Kai-Wei Chang, Wei-Chih Chen, En-Pei Hu, Hung-yi Lee+1

MIT · NTU · NTU AI-CoRE

Spoken Dialogue Systems Instruction Following Speech Generation Evaluation

Open paper AskPDF arXiv

We introduce TiCo, a time-controllable spoken dialogue model (SDM) that follows time-constrained instructions (e.g., "Please generate a response lasting about 15 seconds") and generates spoken responses with controllable duration. This capability is valuable for real-world spoken language systems such as voice assistants and interactive agents, where controlling response duration can improve interaction quality. However, despite their strong ability to generate natural spoken responses, existing models lack time awareness and struggle to follow duration-related instructions. To systematically evaluate this, we introduce TiCo-Bench, the first benchmark for time-controllable instruction following in SDMs, on which existing open-source and commercial models frequently fail to satisfy explicit time constraints. TiCo addresses this limitation by enabling an SDM to estimate elapsed speaking time during generation through Spoken Time Markers (STM) (e.g., <10.6 seconds>). These markers help the model maintain awareness of time and adjust the remaining content to meet the target duration. TiCo is post-trained efficiently without question-answer paired data, relying on self-generation and reinforcement learning with verifiable reward. Experimental results show that TiCo reduces duration error by 2.7x over its backbone and 1.6x over the strongest baseline, while preserving response quality.

Abstract

Spoken Dialogue Systems

Explore similar work

TiCo: Time-Controllable Spoken Dialogue Model

Full-duplex Spoken Language Modeling with Discrete Diffusion

FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation