Modern Text-to-Speech (TTS) systems increasingly leverage Large Language Model (LLM) architectures to achieve scalable, high-fidelity, zero-shot generation. However, these systems typically rely on fixed-frame-rate acoustic tokenization, resulting in speech sequences that are significantly longer than, and asynchronous with their corresponding text. Beyond computational inefficiency, this sequence length disparity often triggers hallucinations in TTS and amplifies the modality gap in spoken language modeling (SLM). In this paper, we propose a novel tokenization scheme that establishes one-to-one synchronization between continuous acoustic features and text tokens, enabling unified, single-stream modeling within an LLM. We demonstrate that these synchronous tokens maintain high-fidelity audio reconstruction and can be effectively modeled in a latent space by a large language model with a flow matching head. Moreover, the ability to seamlessly toggle speech modality within the context enables text-only guidance--a technique that blends logits from text-only and text-speech modes to flexibly bridge the gap toward text-only LLM intelligence. Experimental results indicate that our approach achieves performance competitive with state-of-the-art TTS and SLM systems while virtually eliminating content hallucinations and preserving linguistic integrity, all at a significantly reduced inference cost.
Figures & tables
Figure 1: Word-level alignment via Viterbi decoding. An illustration of the forced alignment between a speech waveform and the transcript “That’s exactly what happened…”. The Viterbi algorithm identifies the most probable frame-level assignment to determine a position pi for each token wi in the encoded text sequence.
Figure 2: Operating under a Variational Autoencoder (VAE) framework, our model utilizes a symmetric encoder-decoder architecture. Each module integrates a CNN-based component for local acoustic feature extraction and reconstruction, complemented by a transformer-based backbone designed to capture the dynamic temporal range of synchronized speech-text sequences.
Figure 3: Attention mask of the encoder (left) and the decoder (right). Asterisks ( ∗ ) mark text-assigned temporal indices. In the encoder, non-text-assigned positions are restricted to intra-block attention, excluding boundary tokens; conversely, text-assigned positions are permitted to attend across both preceding and succeeding blocks. The decoder also utilizes a localized mechanism where each position attends to the current and immediately preceding blocks.
Figure 4: Each text token wi is paired with the speech representation at the K -shifted position, comprising token features si−K , the number of preceeding frames fi−Kbefore , the number of successive frames fi−Kafter , and processed by an autoregressive decoder. The decoder predicts the next text token and produces a conditioning vector, which the flow matching head uses to generate the next speech representation (si−K+1,fi−K+1before,fi−K+1before) .
Models
FPS
BPF
CER ↓
SIM ↑
oMOS ↑
Reference
24k
384
0.00
85.4
3.17
Encodec [ 3 , 55 ]
75
40
0.16
82.4
3.04
Encodec [ 3 , 55 ]
75
80
0.11
83.6
3.10
DAC-24khz [ 4 ]
75
80
0.17
83.4
2.94
Mimi [ 5 ]
12.5
352
0.14
83.9
3.04
Higgs V2 [ 6 ]
25
80
0.10
84.1
3.20
Table 1: Comparison of reconstruction performance across discrete and continuous tokenizers. We report the frame rate (fps), bits per frame (bpf), and key performance metrics (CER, SIM, and oMOS). Continuous baselines ( † ) are reported at 16-bit precision.
SeedTTS-Eval
LibriTTSR-Eval
Models
Mem
RTF
Hours
CER
SIM
oMOS
CER
SIM
oMOS
XTTS v2 [ 25 ]
2.1
0.19
30K
2.10
70.3
2.67
0.59
74.0
3.17
Index-TTS2 [ 17 ]
10.1
0.58
55K
0.31
79.8
2.95
0.23
83.3
3.34
Higgs Audio v2 [ 6 ]
16.3
0.44
10M
9.57
75.3
2.98
1.88
79.7
3.28
VibeVoice 1.5B [ 19 ]
5.2
0.51
0.5–1.6M
3.07
72.3
2.92
1.25
79.5
3.24
FireRedTTS-2 [ 18 ]
12.2
0.76
1.1–1.4M
0.81
75.1
2.96
1.44
81.2
3.28
Table 2: Voice cloning evaluation. Objective comparison between TADA and baseline models using character error rate (CER), speaker similarity (SIM) and Objective Mean Opinion Score (oMOS) metrics.
Models
CER
SIM
oMOS
sSIM
sMOS
Index-TTS [ 17 ]
1.90
76.9
2.84
4.25
3.61
VibeVoice 1.5B [ 19 ]
2.51
73.3
2.54
3.92
3.91
FireRedTTS-2 [ 18 ]
21.6
73.8
2.84
3.98
3.58
TADA-3B
2.34
67.0
2.86
-
-
+ Text-Free Guidance
4.30
72.4
2.87
-
-
+ Online RS
2.74
74.7
2.84
4.18
3.78
Table 3: Performance comparison for long-form, expressive speech generation using EARS dataset samples.
SI PPL ↓
sSC ↑
tSC ↑
Models
T
TS
T
TS
T
TS
Llama-1B-Ins
23.7
-
73.8
-
97.4
-
Llama-3B-Ins
20.9
-
79.4
-
98.0
-
TWIST-7B [ 54 ]
-
-
-
55.3
-
74.1
TWIST-13B [ 54 ]
-
-
-
55.4
-
76.4
SpiritLM-7B [ 26 ]
-
-
79.4
61.0
98.0
82.9
Table 4: Language capability evaluation on the Seamless Interaction (SI), sSC, and tSC datasets, comparing text-only (T) and text-speech (TS) inference modes with and without Text-Only Guidance.
Figure 5: Reconstruction quality (CER, SS, and oMOS) vs. token density for uniformly spaced tokens.
Nfl
RTF
TLLMstep
TFMstep
CER
SIM
oMOS
2 steps
0.05
31
8
1.82
77.3
2.82
4 steps
0.05
31
14
0.85
79.1
2.96
10 steps
0.09
31
23
0.55
80.2
3.11
20 steps
0.13
31
54
0.63
79.8
3.11
Table 5: Analysis of inference latency and TTS performance of TADA-1B across varying flow matching sampling steps.
Models
PPL
CER
SIM
oMOS
Llama-1B-Instruct
23.7
-
-
-
TADA-1B ( λCE=0.1 , Base)
36.5
0.87
78.4
3.02
TADA-1B ( λCE=1.0 )
31.2
0.89
77.6
3.00
TADA-1B ( λCE=λKD=0.1 )
26.6
0.84
77.7
2.98
Table 6: Abalation results of TADA-1B with different language preservation loss config.
School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China · DiDi Global Inc., Beijing, China