CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models
Authors: Yingzhuo Deng, Chong Li, Bingxin Liu, Yi Shu, Tianyu Peng, Can Xie, Siyao Song, Shuo Ren, Jiajun Zhang
Organizations: Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Wuhan AI Research
Spoken Dialogue Models (SDMs) typically share a speech-token space for generation and synthesis, leading to long autoregressive sequences and costly inference. Lower-rate speech tokenizers shorten sequences but may introduce compression-quality trade-offs and costly replacement without breaking this coupling. To address this limitation, we propose CoSTA, a compatibility-preserving post-hoc framework that decouples generation and synthesis token spaces while retaining the original speech tokenizer and decoder. CoSTA extends the generation token space with compressed representations via mean-initialized vocabulary extension, pronunciation adaptation, and dialogue adaptation, followed by deterministic expansion back to the original speech-token space before synthesis. On bilingual speech-dialogue benchmarks, CoSTA achieves approximately 1.7× expanded speech-token generation speed, lower full-response latency, and higher throughput while preserving response quality. Analyses show more sustained interaction under limited budgets and a practical trade-off among compression, decoding efficiency, and quality. Further experiments demonstrate cross-architecture transfer and complementarity to low-rate tokenization.
Figures & tables
Figure 1: Overview of CoSTA. Existing SDMs couple generation and synthesis token spaces. CoSTA extends the generation token space with compressed representations and deterministically expands them back to the original space before synthesis.
Figure 2: Speech-token BPE scaling across eight tokenizer configurations. Compression rate decreases approximately linearly with log2 vocabulary size.
Figure 3: CoSTA combines vocabulary extension with mean-based initialization, progressive pronunciation adaptation, and dialogue adaptation.
Figure 4: Inference with deterministic expansion from the extended generation token space to the original space before unchanged decoding.
Model
AlpacaEval-zh
APE-zh
Claude-zh
LCSTS-zh
MLC-zh
Wildchat-zh
Avg
ρg↓
Speed ↑
ρg↓
Speed ↑
ρg↓
Speed ↑
ρg↓
Speed ↑
ρg↓
Speed ↑
ρg↓
Speed ↑
ρg↓
Speed ↑
Relative ↑
Step-Audio 2 mini
100.00%
55.51
100.00%
60.88
100.00%
56.39
100.00%
57.05
100.00%
65.22
100.00%
57.40
100.00%
58.74
100.00%
Vanilla-FT-ZH
100.00%
57.84
100.00%
60.40
100.00%
54.99
100.00%
55.54
100.00%
63.47
100.00%
59.10
100.00%
58.56
99.69%
Vanilla-FT-MIX
100.00%
59.01
100.00%
59.65
100.00%
56.93
100.00%
56.03
100.00%
61.57
100.00%
59.08
100.00%
58.71
99.95%
CoSTA-ZH
67.50%
103.03
64.63%
101.68
68.52%
101.20
69.40%
94.46
62.70%
110.31
68.13%
101.29
66.81%
101.99
173.63%
CoSTA-MIX
69.12%
98.44
64.68%
102.34
70.58%
92.54
71.06%
91.37
63.03%
106.32
70.21%
93.22
68.11%
97.37
165.76%
Table 1: Model-side efficiency on Chinese and English speech-to-speech benchmarks. ρg denotes generated-token compression, Speed denotes expanded speech-token generation speed, and Relative normalizes Speed to Step-Audio 2 mini within each language setting.
Concurrency
Model
Full-Response Latency (s) ↓
Throughput (req/s) ↑
Peak Memory (GiB)
1
Step-Audio 2 mini / Vanilla-FT-ZH / CoSTA-ZH
10.73 / 9.72 / 7.14
0.103 / 0.114 / 0.160
78.12 / 77.64 / 78.56
4
Step-Audio 2 mini / Vanilla-FT-ZH / CoSTA-ZH
13.63 / 11.38 / 9.16
0.316 / 0.369 / 0.479
78.34 / 78.13 / 79.02
8
Step-Audio 2 mini / Vanilla-FT-ZH / CoSTA-ZH
15.56 / 13.71 / 9.49
0.515 / 0.601 / 0.820
78.90 / 79.17 / 79.14
1
Step-Audio 2 mini / Vanilla-FT-EN / CoSTA-EN
12.50 / 16.16 / 9.25
0.088 / 0.068 / 0.122
78.98 / 77.16 / 78.69
4
Step-Audio 2 mini / Vanilla-FT-EN / CoSTA-EN
15.52 / 20.17 / 11.59
0.276 / 0.212 / 0.372
79.24 / 78.33 / 79.12
8
Step-Audio 2 mini / Vanilla-FT-EN / CoSTA-EN
18.62 / 25.04 / 14.44
0.442 / 0.330 / 0.556
78.52 / 77.73 / 79.04
Table 2: System-level efficiency on AlpacaEval-zh (upper) and AlpacaEval (lower) at concurrency 1, 4, and 8 on a single A100 80GB GPU. Values are ordered as Step-Audio 2 mini, Vanilla-FT, and CoSTA.
Model
AlpacaEval-zh
APE-zh
Claude-zh
LCSTS-zh
MLC-zh
Wildchat-zh
Avg
G-Eval ↑
ASR-CER ↓
UTMOS ↑
Qwen2.5-Omni
74.10
62.11
78.86
69.75
82.53
78.35
74.28
2.93%
2.95
Kimi-Audio
77.05
39.12
80.24
76.13
80.92
78.08
71.93
4.88%
2.52
GLM-4-Voice
84.76
23.16
83.54
76.36
78.85
84.41
71.85
3.77%
2.94
Step-Audio 2 mini
80.18
21.93
72.40
79.16
78.16
83.59
69.24
7.49%
3.62
Vanilla-FT-ZH
79.77
21.40
80.35
78.82
77.47
81.47
69.88
4.96%
3.77
Table 3: Automatic quality evaluation on Chinese and English speech-to-speech benchmarks. Benchmark columns report G-Eval; Avg reports average G-Eval, ASR-CER/WER, and UTMOS. Bold and underline mark the best and second-best results among Step-Audio 2 mini variants; external models serve as quality references.
Figure 5: Paired human evaluation of CoSTA against the base model and Vanilla-FT. Bars show wins, ties, and losses from 180 paired MOS ratings per language and metric.
Variant
ρg↓
ASR-CER ↓
UTMOS ↑
G-Eval ↑
Step-Audio 2 mini
100.00%
7.49%
3.62
69.24
Vanilla-FT-ZH
100.00%
4.96%
3.77
69.88
CoSTA-ZH
66.73%
1.51%
3.75
72.27
w/o Timbre Conversion
67.01%
1.52%
3.75
70.93
w/o Progressive Adaptation
67.75%
1.46%
3.75
72.19
w/o Pronunciation Adaptation
69.56%
1.74%
3.74
72.12
Table 4: Component ablation on Chinese benchmarks. ρg , ASR-CER, UTMOS, and G-Eval measure generated-token compression, text-speech alignment, speech quality, and response-content quality, respectively.
Figure 6: Training loss during the first stage of pronunciation adaptation. Mean initialization achieves faster early convergence than the alternatives.
Figure 7: Vocabulary-size trade-off on AlpacaEval-zh: relative expanded speech-token generation speed with (a) UTMOS and (b) G-Eval. The 0K baseline denotes the original Step-Audio 2 mini; other points denote CoSTA-ZH.
Table 6: Transfer of CoSTA to OpenS2S. Upper/lower blocks report Chinese/English results using ASR-CER/WER, respectively. Matched Vanilla-FT uses the same architecture and language-specific adaptation data.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Data split
50% / 50%
Stage-1 trainable params
Embedding, LM head
Stage-2 trainable params
+ Intermediate layers
Optimizer
AdamW
Stage-1 learning rate
1.6×10−4
Stage-2 learning rate
2.0×10−5
Appendix
Table 7: Hyperparameters used during pronunciation adaptation.
Hyperparameter
Value
Trainable params
All LLM parameters
Optimizer
AdamW
Learning rate
2.0×10−5
Scheduler
Cosine
Warmup ratio
0.03
Global batch size
256
Appendix
Table 8: Hyperparameters used during dialogue adaptation.
Hyperparameter
Value
Decoding mode
Greedy
Repetition penalty
1.2
Max new tokens
2048
BPE expansion
Enabled
Expansion target
Speech-token stream only
Text-token stream
Unchanged
Appendix
Table 9: Hyperparameters used during inference.
Data Type
Language
Scale
Text-to-speech
Chinese
300h
English
300h
Speech dialogue
Chinese
49,664 examples
English
49,408 examples
Appendix
Table 10: Statistics of training data.
Dataset
#Samples
Task / Aspect
Eval. Mode
AlpacaEval-zh
147
Authentic, open-ended dialogue
Open
AlpacaEval
199
Authentic, open-ended dialogue
Open
APE-zh
190
Practical mathematical problems
QA
Gsm8kEval
582
Practical mathematical problems
QA
Claude-zh
222
Authentic, open-ended dialogue
Open
CommonEval
200
Authentic, open-ended dialogue
Open
Appendix
Table 11: URO-Bench datasets used in our evaluation. The selected datasets cover understanding, reasoning, and oral conversation in Chinese and English speech-to-speech settings.
Figure 8: Gradient norm during the first stage of pronunciation adaptation. Mean initialization yields more stable optimization than the alternatives.
Figure 9: Held-out compression across data sources and languages with S3Tokenizer V2. Points are test measurements; lines are fits against log added vocabulary size. Emilia-TC denotes timbre-converted Emilia.
Figure 10: Training-corpus compression across eight tokenizer configurations on Chinese dialogue speech, corresponding to the test results in Figure 2 . Points are measurements and lines are log-linear fits. Mimi and Qwen3-TTS use their first semantic codebook.
Tokenizer
Public reference source
Codes
Hz
Token stream used for BPE
GLM-4-Voice Tokenizer 12.5Hz
GLM-4-Voice tokenizer
16,384
12.5
Single quantized speech stream
S3Tokenizer V2 25Hz
CosyVoice2-0.5B
6,561
25
Single semantic-token stream
S3Tokenizer V3 25Hz
Fun-CosyVoice3-0.5B-2512
6,561
25
Single semantic-token stream
Mimi
kyutai/mimi
2,048
12.5
First semantic codebook only
S3Tokenizer V1 25Hz
CosyVoice-300M-25Hz
4,096
25
Single semantic-token stream
Qwen3-TTS-Tokenizer-12Hz
Qwen3-TTS tokenizer
2,048
12.5
First semantic codebook only
Appendix
Table 12: Tokenizer configurations for Chinese dialogue compression, using the figure names and linked public sources. Codes excludes added BPE units. Hz is the native rate; mHuBERT is omitted. Qwen3-TTS operates at 12.5 Hz despite its model name ( official configuration ).
Figure 11: Training-corpus compression across data sources and languages with S3Tokenizer V2, corresponding to the test results in Figure 9 . Points are measurements and lines are log-linear fits. Each vocabulary is learned on the training corpus represented by its curve.
Name
License / Terms
Step-Audio 2 mini
Apache 2.0 license
Qwen2.5-Omni
Apache 2.0 license
Kimi-Audio
Apache 2.0 / MIT licenses
GLM-4-Voice / Tokenizer
glm-4-voice Model License
OpenS2S
Apache 2.0 license
Emilia
CC BY-NC 4.0 license
Appendix
Table 13: Licenses and usage terms of scientific artifacts used in this work.
Model
AlpacaEval-zh
APE-zh
Claude-zh
Avg
CER ↓
UTMOS ↑
G-Eval ↑
CER ↓
UTMOS ↑
G-Eval ↑
CER ↓
UTMOS ↑
G-Eval ↑
CER ↓
UTMOS ↑
G-Eval ↑
Qwen2.5-Omni
2.09%
2.93
74.10
8.50%
3.02
62.11
2.47%
2.73
78.86
2.93%
2.95
74.28
Kimi-Audio
4.71%
2.52
77.05
8.29%
2.59
39.12
11.12%
2.08
80.24
4.88%
2.52
71.93
GLM-4-Voice
3.65%
2.86
84.76
5.51%
3.14
23.16
7.13%
2.59
83.54
3.77%
2.94
71.85
Step-Audio 2 mini
8.48%
3.58
80.18
4.35%
3.87
21.93
20.28%
3.33
72.40
7.49%
3.62
69.24
Vanilla-FT-ZH
5.11%
3.76
79.77
0.95%
4.06
21.40
19.51%
3.41
80.35
4.96%
3.77
69.88
Appendix
Table 14: Detailed quality results on Chinese speech-to-speech benchmarks. Each benchmark reports CER, UTMOS, and G-Eval for text-speech alignment, speech quality, and response-content quality, respectively.
Model
AlpacaEval
Gsm8kEval
CommonEval
Avg
WER ↓
UTMOS ↑
G-Eval ↑
WER ↓
UTMOS ↑
G-Eval ↑
WER ↓
UTMOS ↑
G-Eval ↑
WER ↓
UTMOS ↑
G-Eval ↑
Qwen2.5-Omni
2.05%
4.22
76.78
4.25%
4.32
54.30
1.48%
4.27
72.00
1.73%
4.28
72.88
Kimi-Audio
20.77%
2.53
62.01
99.08%
2.77
0.57
37.62%
2.95
50.77
34.92%
2.86
50.46
GLM-4-Voice
13.92%
4.00
80.87
8.29%
4.25
29.32
26.28%
4.10
66.77
11.55%
4.16
69.95
Step-Audio 2 mini
9.82%
4.51
76.31
17.07%
4.53
15.92
1.64%
4.53
52.53
6.49%
4.53
63.73
Vanilla-FT-EN
23.82%
4.35
73.74
4.15%
4.53
16.67
9.73%
4.47
72.03
10.16%
4.46
66.59
Appendix
Table 15: Detailed quality results on English speech-to-speech benchmarks. Each benchmark reports WER, UTMOS, and G-Eval for text-speech alignment, speech quality, and response-content quality, respectively.
Model
TTS Data
AlpacaEval-zh
APE-zh
Avg
ρg↓
ASR-CER ↓
UTMOS ↑
G-Eval ↑
ρg↓
ASR-CER ↓
UTMOS ↑
G-Eval ↑
ρg↓
ASR-CER ↓
UTMOS ↑
G-Eval ↑
Step-Audio 2 mini
–
100.00%
8.48%
3.54
79.95
100.00%
4.28%
3.87
22.11
100.00%
8.57%
3.66
63.34
CoSTA-ZH
500h
68.17%
1.62%
3.70
80.86
65.32%
1.11%
4.06
21.05
66.48%
1.52%
3.79
66.35
400h
68.73%
1.69%
3.70
82.40
65.88%
1.48%
4.05
25.26
67.35%
1.62%
3.79
67.71
300h
68.54%
1.91%
3.72
81.59
65.09%
1.02%
4.06
25.09
66.83%
1.66%
3.78
67.75
200h
69.29%
2.15%
3.68
82.36
65.69%
0.83%
4.07
26.84
67.24%
1.71%
3.78
67.99
Appendix
Table 16: Effect of pronunciation-adaptation data scale on Chinese benchmarks. Dialogue-adaptation data is fixed to 50K examples. ρg denotes the generated-token compression rate, while ASR-CER, UTMOS, and G-Eval evaluate text-speech alignment, speech quality, and response-content quality, respectively.
Model
Dialogue Data
AlpacaEval-zh
APE-zh
Avg
ρg↓
ASR-CER ↓
UTMOS ↑
G-Eval ↑
ρg↓
ASR-CER ↓
UTMOS ↑
G-Eval ↑
ρg↓
ASR-CER ↓
UTMOS ↑
G-Eval ↑
Step-Audio 2 mini
–
100.00%
8.48%
3.54
79.95
100.00%
4.28%
3.87
22.11
100.00%
8.57%
3.66
63.34
CoSTA-ZH
50K
68.54%
1.91%
3.72
81.59
65.09%
1.02%
4.06
25.09
66.83%
1.66%
3.78
67.75
40K
69.24%
1.79%
3.71
78.59
65.61%
1.36%
4.04
22.81
67.28%
1.72%
3.78
66.02
30K
68.82%
1.90%
3.73
79.23
66.04%
1.00%
4.04
18.07
67.50%
1.80%
3.78
64.95
20K
68.32%
2.02%
3.70
80.45
66.20%
0.95%
4.06
25.44
67.49%
1.97%
3.79
66.84
Appendix
Table 17: Effect of dialogue-adaptation data scale on Chinese benchmarks. Pronunciation-adaptation data is fixed to 300h. ρg denotes the generated-token compression rate, while ASR-CER, UTMOS, and G-Eval evaluate text-speech alignment, speech quality, and response-content quality, respectively.
Model
ρg↓
Speed ↑
Prefill (ms) ↓
Decode (ms) ↓
LLM Total (ms) ↓
Speech Decoder (ms) ↓
Total (ms) ↓
Step-Audio 2 mini
100.00%
58.74
56.02
164.42
220.44
337.46
557.90
Vanilla-FT-ZH
100.00%
58.56
55.37
164.22
219.59
340.51
560.10
CoSTA-ZH
66.81%
101.99
54.55
122.98
177.53
328.06
505.59
Step-Audio 2 mini
100.00%
59.69
53.07
165.08
218.15
344.44
562.60
Vanilla-FT-EN
100.00%
57.67
55.72
165.10
220.82
339.52
560.34
CoSTA-EN
70.17%
99.05
53.24
115.96
169.20
339.17
508.36
Appendix
Table 18: First-packet latency breakdown for Chinese and English settings. The upper and lower blocks report Chinese and English results, respectively.
Metric
Model
AlpacaEval-zh
APE-zh
Claude-zh
LCSTS-zh
MLC-zh
Wildchat-zh
Avg.
Speech Fluency
Step-Audio 2 mini
4.70
4.77
3.77
4.47
4.83
4.37
4.48
Vanilla-FT-ZH
3.80
4.50
3.60
4.83
4.60
4.77
4.35
CoSTA-ZH
4.73
4.33
4.60
4.67
4.70
4.73
4.63
Semantic Quality
Step-Audio 2 mini
4.17
2.23
3.17
3.60
3.83
4.10
3.52
Vanilla-FT-ZH
4.00
1.83
3.43
3.80
3.70
4.23
3.50
CoSTA-ZH
4.40
2.17
3.60
4.10
4.00
4.23
3.75
Appendix
Table 19: MOS-style human evaluation on Chinese URO-Bench datasets. Scores are averaged over six evaluators and five randomly sampled queries per dataset. Higher scores indicate better performance.
Metric
Model
AlpacaEval
Gsm8kEval
CommonEval
Summary
MLC
WildchatEval
Avg.
Speech Fluency
Step-Audio 2 mini
4.23
4.03
4.73
4.70
4.50
4.80
4.50
Vanilla-FT-EN
3.53
4.63
4.23
4.77
4.30
4.37
4.31
CoSTA-EN
4.80
4.70
4.77
4.87
4.83
4.73
4.78
Semantic Quality
Step-Audio 2 mini
3.50
2.33
1.23
4.57
4.20
4.27
3.35
Vanilla-FT-EN
3.13
1.77
4.00
4.40
4.80
4.27
3.73
CoSTA-EN
4.37
1.20
4.17
4.90
4.83
4.73
4.03
Appendix
Table 20: MOS-style human evaluation on English URO-Bench datasets. Scores are averaged over six evaluators and five randomly sampled queries per dataset. Higher scores indicate better performance.
Model
AlpacaEval-zh
Claude-zh
LCSTS-zh
Avg
ρg↓
ASR-CER ↓
UTMOS ↑
G-Eval ↑
ρg↓
ASR-CER ↓
UTMOS ↑
G-Eval ↑
ρg↓
ASR-CER ↓
UTMOS ↑
G-Eval ↑
ρg↓
ASR-CER ↓
UTMOS ↑
G-Eval ↑
OpenS2S
100.00%
9.02%
2.54
69.71
100.00%
14.40%
2.18
66.97
100.00%
9.82%
2.51
59.10
100.00%
11.08%
2.41
65.26
Vanilla-FT-ZH
100.00%
6.26%
2.49
72.29
100.00%
20.00%
2.14
66.64
100.00%
7.81%
2.50
63.25
100.00%
11.36%
2.38
67.39
CoSTA-ZH
78.34%
5.98%
2.52
70.61
79.92%
11.77%
2.08
71.65
78.23%
7.75%
2.49
65.27
78.83%
8.50%
2.36
69.18
Model
AlpacaEval
CommonEval
Summary
Avg
ρg↓
ASR-WER ↓
UTMOS ↑
G-Eval ↑
ρg↓
ASR-WER ↓
UTMOS ↑
G-Eval ↑
ρg↓
ASR-WER ↓
UTMOS ↑
G-Eval ↑
ρg↓
ASR-WER ↓
UTMOS ↑
G-Eval ↑
Appendix
Table 21: Detailed SDM-level generalization results on OpenS2S. CoSTA is compared with the original OpenS2S model and matched vanilla fine-tuning on Chinese and English benchmarks. ρg denotes the generated-token compression rate, while ASR-CER (Chinese) / ASR-WER (English), UTMOS, and G-Eval evaluate text-speech alignment, speech quality, and response-content quality, respectively.
Model
Original
BPE
Original/BPE
BPE coverage
Avg. expansion
CoSTA-ZH
70.95%
29.05%
2.44
51.64%
2.608
w/o Dialogue Adaptation
78.95%
21.05%
3.75
36.71%
2.175
Appendix
Table 22: Speech-token usage with and without dialogue adaptation on all six Chinese benchmarks (1,122 responses per model). Statistics are pooled over generated speech tokens. (a) Original/BPE shares and their ratio use counts before expansion; BPE coverage uses counts after expansion. Avg. expansion is the number of original tokens per generated BPE unit. (b) L is the expansion length, with L=1 denoting an original token. Each percentage uses all generated speech tokens before expansion as its denominator.
Model
Size
ρg↓
Speed ↑
Relative ↑
UTMOS ↑
G-Eval ↑
Step-Audio 2 mini
0K
100.00%
55.51
100.00%
3.58
80.18
CoSTA-ZH
10K
76.66%
78.67
141.71%
3.73
80.86
20K
74.33%
84.51
152.24%
3.71
80.73
40K
71.25%
93.20
167.89%
3.69
83.35
80K
67.50%
103.03
185.59%
3.70
81.36
160K
63.62%
104.18
187.65%
3.68
78.87
Appendix
Table 23: Complete vocabulary-size results on AlpacaEval-zh underlying Figure 7 . Size is the number of added BPE units; Speed is expanded speech tokens/s, normalized to the original Step-Audio 2 mini in Relative. The 0K row is the unadapted baseline.
Added units
Compression ↓
Median ms/step ↓
Overhead ↓
Norm. Cost ↓
0K
100.00%
0.6902
+0.00%
1.000
10K
73.27%
0.7320
+6.06%
0.777
20K
69.79%
0.7739
+12.13%
0.783
40K
66.15%
0.8576
+24.25%
0.822
80K
62.29%
1.0251
+48.52%
0.925
160K
58.48%
1.3598
+97.01%
1.152
Appendix
Table 24: Complete corpus-compression and isolated LM-head projection plus greedy-argmax costs for 0K–320K added BPE units, measured on a single A100 GPU with identical BF16 hidden states at batch size 1. Compression uses held-out Dialogue-ZH token counts. Derived costs use unrounded compression rates. Overhead is per-step latency relative to 0K; Norm. Cost is ρMtM/t0 for a fixed expanded token count (Equation 9 ).
Tokenizer
Split
α (%)
β (pp/doubling)
R2
RMSE (pp)
Max ∣e∣ (pp)
GLM-4-Voice Tokenizer 12.5Hz
Train
84.884
-4.808
0.9959
0.528
0.784
Test
84.866
-4.193
0.9989
0.242
0.386
S3Tokenizer V2 25Hz
Train
73.550
-4.013
0.9988
0.241
0.370
Test
73.397
-3.698
0.9997
0.103
0.149
S3Tokenizer V3 25Hz
Train
71.678
-4.107
0.9999
0.085
0.138
Test
71.569
-3.809
0.9991
0.191
0.309
Appendix
Table 25: Complete training and test fits for the eight tokenizer configurations on Chinese dialogue speech. Full names, configurations, and public reference sources are given in Table 12 . Fits use y=α+βlog2(M/10K) , where y=100ρ . Each row fits six vocabulary settings; errors are in percentage points.
Setting
Split
α (%)
β (pp/doubling)
R2
RMSE (pp)
Max ∣e∣ (pp)
Emilia-MIX
Train
85.902
-3.653
0.9929
0.529
0.752
Test
85.664
-3.621
0.9942
0.474
0.670
Emilia-EN
Train
85.513
-3.706
0.9942
0.485
0.693
Test
85.127
-3.680
0.9957
0.414
0.605
Emilia-ZH
Train
84.656
-3.751
0.9947
0.468
0.676
Test
84.464
-3.681
0.9954
0.426
0.618
Appendix
Table 26: Complete training and test fits across data sources and languages with S3Tokenizer V2. Fits use y=α+βlog2(M/10K) , where y=100ρ . Each row fits six vocabulary settings; errors are in percentage points.
We introduce TiCo, a time-controllable spoken dialogue model (SDM) that follows time-constrained instructions (e.g., "Please generate a response lasting about 15 seconds") and generates spoken responses with controllable duration. This capability is valuable for real-world spoken language systems such as voice assistants and interactive agents, where controlling response duration can improve interaction quality. However, despite their strong ability to generate natural spoken responses, existing models lack time awareness and struggle to follow duration-related instructions. To systematically evaluate this, we introduce TiCo-Bench, the first benchmark for time-controllable instruction following in SDMs, on which existing open-source and commercial models frequently fail to satisfy explicit time constraints. TiCo addresses this limitation by enabling an SDM to estimate elapsed speaking time during generation through Spoken Time Markers (STM) (e.g., <10.6 seconds>). These markers help the model maintain awareness of time and adjust the remaining content to meet the target duration. TiCo is post-trained efficiently without question-answer paired data, relying on self-generation and reinforcement learning with verifiable reward. Experimental results show that TiCo reduces duration error by 2.7x over its backbone and 1.6x over the strongest baseline, while preserving response quality.
We introduce Diffuplex, a full-duplex spoken language model based on discrete diffusion that achieves an unprecedented balance between linguistic intelligence and response efficiency. With thinking mode, Diffuplex delivers 13.0 tokens per forward pass, reaching an average of 1872 tokens per second and substantially surpassing the frontier autoregressive (AR) models in decoding speed. Without thinking, Diffuplex matches the first-audio latency of existing full-duplex models while achieving markedly stronger linguistic intelligence. This capability is enabled by separating when to say from what to say at the training level, rather than compressing both into a single autoregressive timeline. During inference, Diffuplex continuously tracks the user speech through an AR interaction policy, while independently generating and refining response content through parallel diffusion. Consequently, semantic computation is no longer bounded by the perception audio frame rate, allowing intelligence and responsiveness to improve simultaneously. Taken together, these results establish a new paradigm for spoken agents that can sustain real-time dialogue while devoting parallel computation to intelligent response generation. Training code and checkpoints will be avaliable soon.
Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens. Existing speech-language models project high-frame-rate acoustic features directly into the LLM input space, making long-context processing computationally prohibitive. Unlike images or videos, speech lacks spatial redundancy, making extreme token compression particularly challenging. To address this limitation, we propose FastSLM, a token-efficient architecture featuring the Hierarchical Temporal Abstractor (HTA), which progressively distills acoustic features across multiple temporal scales. HTA achieves an extreme compression rate of 1.67 tokens per second (97% reduction) while preserving essential linguistic information for downstream speech-language understanding. Experimental results demonstrate that FastSLM achieves competitive performance across diverse speech-language tasks while requiring substantially fewer speech tokens and FLOPs than existing speech-language models. The source code and model checkpoints are available at https://github.com/Lee-junseok1025/FastSLM.