Streaming Inverse Text Normalization (ITN) is vital for converting spoken-form outputs from streaming Automatic Speech Recognition into formatted written text. Existing streaming ITN methods rely on hybrid systems combining neural tagging with expert-crafted finite-state transducer rules, limiting scalability across domains and languages. While end-to-end models offer superior scalability by learning directly from data, standard encoder-decoder architectures are inherently non-streaming due to global attention. In this paper, we propose an efficient streaming end-to-end ITN system, adapting a pretrained text-to-text model to leverage its robust linguistic knowledge. To enable streaming, we introduce architectural adaptations, a specialized training strategy, a Read-Tag-Write decoding policy, and inference optimizations. Experiments on Vietnamese datasets show accuracy comparable to non-streaming baselines, outperforming hybrid methods while satisfying real-time latency requirements.
Figures & tables
Figure 1: Overview of the proposed streaming end-to-end ITN framework using the Read–Tag–Write policy. The streaming encoder utilizes Dynamic Right Context masking to process incoming chunks of spoken-form tokens, while a joint tagger predicts {B, I, O} labels. Verbatim tokens (O) are passed through instantly to minimize latency. Upon the completion of a normalization span, the autoregressive decoder is selectively triggered, generating normalized text by cross-attending to the relevant encoder hidden states ( H0:m ) and conditioning on the written-form prefix y^pre .
With PnC
Without PnC
Model
Pretrained
c
r
F 1
I-WER
NI-WER
WER
I-WER
NI-WER
WER
Non-stream
B0
×
∞
∞
–
14.20
1.51
4.61
8.28
0.54
1.11
B1
EnViT5
∞
∞
–
11.69
1.30
3.84
5.17
0.39
0.74
B2
EnViT5
∞
∞
97.34
10.80
1.32
3.63
5.03
0.16
0.52
AdapITN [ 2 ]
EnViBert
∞
∞
96.65
156.88
2.50
40.08
14.58
0.40
1.45
SC (ours)
EnViT5
{3,4,5}
{0,1,2}
96.62
11.48
1.31
3.79
5.83
0.18
0.60
Table 1: Evaluation of ITN performance with and without punctuation and capitalization (PnC). Parameters c and r denote chunk and right-context sizes during inference. All metrics are reported as percentages (%). Model categories: B (Non-streaming baselines), H (Hybrid streaming models), and S (Proposed end-to-end streaming variants).
Figure 2: Per-chunk latency breakdown (Encoder vs. Total) for four streaming configurations across chunk sizes of 3, 4, and 5.
Model
I-WER
NI-WER
WER
Non-stream
B0
37.30
2.44
10.94
B1
34.09
2.24
10.01
B2
35.33
2.28
10.34
SC
36.18
2.27
10.54
Stream
H2
47.56
2.75
13.67
Wait- k
41.16
2.48
11.91
Table 2: Real-world ITN performance on noisy ASR transcripts. All metrics are reported as percentages (%).
Figure 3: Impact of truncation probability ptrunc on the Word Error Rate for both offline and streaming inference of our model.
Lang.
Backbone
Params
Stream
WER (%)
ASR-L
ASR-O
Vi
BartPho
132M
×
4.61
11.07
✓
6.31
12.84
EnViT5
275M
×
3.84
10.01
✓
4.54
11.02
W/ PnC
W/o PnC
Table 3: Generalizability of ITN performance across architectures and languages. Values represent average WER (%) across c∈{3,4,5} and r∈{0,1,2} . Vietnamese evaluation uses ASR-L (ground-truth spoken text) and ASR-O (noisy ASR transcripts); English evaluation uses With and Without PnC.
Streaming TTS that receives streaming text is essential for interactive systems, yet this scheme faces two major challenges: unnatural prosody due to missing lookahead and long-form collapse due to unbounded context. We propose a prosodic-boundary-aware post-training strategy, adapting a pretrained LLM-based TTS model using weakly time-aligned data. Specifically, the model is adapted to learn early stopping at specified content boundaries when provided with limited future text. During inference, a sliding-window prompt carries forward previous text and speech tokens, ensuring bounded context and seamless concatenation. Evaluations show our method outperforms CosyVoice-Style interleaved baseline in both short and long-form scenarios. In long-text synthesis, especially, it achieves a 66.2% absolute reduction in word error rate (from 71.0% to 4.8%) and increases speaker and emotion similarity by 16.1% and 1.5% relatively, offering a robust solution for streaming TTS with incremental text.
Changsong Liu, Tianrui Wang, Ye Ni +2
Nanyang Technological University, Singapore · Tianjin University, China · Southeast University, China
Text-speech joint spoken language modeling (SLM) aims at natural and intelligent speech-based interactions, but developing such a system may suffer from modality mismatch: speech unit sequences are much longer than text tokens. Prior work reduces this gap with text-aligned tokenization and embedding (TASTE), producing speech tokens that align in lengths with their textual counterparts. However, the dependence on an external ASR system and the use of a non-causal decoder limits streaming use. To address this limitation, we propose TASTE-S, a streamable extension of TASTE suitable for real-time usage. TASTE-S integrates a CTC-based ASR module into the encoder for instant dual-modality encoding. We also redesign the unit decoder to enable on-the-fly decoding. With joint training, we show that TASTE-S matches TASTE's performance while significantly reducing latency. Further investigations reveal that TASTE-S remains robust to transcriptions and enables long-form encoding and decoding.
Liang-Hsuan Tseng, Hung-yi Lee
Graduate Institute of Communication Engineering, National Taiwan University
Streaming text-to-speech synthesis in cascaded LLM-TTS systems still faces latency challenges as most TTS models require full context before initiating generation. We present S5-TTS, a streaming variant of T5-TTS that enables low-latency, word-by-word incremental speech synthesis through encoder-decoder language modeling and monotonic alignment learning. S5-TTS begins generating speech immediately after receiving the first few words, substantially reducing end-to-end response latency. To maintain quality under limited lookahead, we introduce a lookahead-causal masking mechanism with Conv-based auxiliary attention that preserves intelligibility and speaker similarity, and employ interleaved multi-source distillation to further restore naturalness. Experiments show that S5-TTS achieves comparable quality to full-context T5-TTS, supports zero-shot synthesis with high speaker similarity, and significantly reduces end-to-end latency for practical conversational AI systems.