Uni-Stream: A Unified Streaming Speech Language Model for Speech Synthesis and Recognition

Xuan Shi; Chang Zeng; Tiantian Feng; Ye Zhang; Jun Liu; Shih-Heng Wang; Lie Lu; Shrikanth Narayanan

eess.ASOct 23, 2026

Uni-Stream: A Unified Streaming Speech Language Model for Speech Synthesis and Recognition

Authors: Xuan Shi, Chang Zeng, Tiantian Feng, Ye Zhang, Jun Liu, Shih-Heng Wang, Lie Lu, Shrikanth Narayanan

Organizations: University of Southern California, USA · National Institute of Informatics, Japan · Massachusetts Institute of Technology, USA · Dolby Laboratories, USA

Abstract

Unified streaming speech language models aim to support speech recognition and synthesis within a single model, enabling real-time human-computer interaction. However, jointly supporting these two directions of speech-text conversion while preserving task-specific performance remains challenging. We propose Uni-Stream, a unified streaming speech language model built on compact pretrained speech models. Uni-Stream processes interleaved text and speech in a streaming format and achieves performance comparable to task-specific models through two-stage training. We also develop a data annotation pipeline for streaming training and will publicly release the generated transcriptions and word-level timestamps. We further examine Uni-Stream's representations across tasks and modalities and find that speech and text residence in distinct latent feature space even when trained in a unified model architecture, while representations within each modality become more task-dependent in later layers.

Figures & tables

Figure 1: Overview of Uni-Stream. (a) Streaming ASR and time-aligned TTS sequences (frame counts illustrative). (b) Shared backbone with text and diffusion heads. (c) Two-stage training: task-specific streaming adaptation followed by joint training. The distillation branches show the objectives evaluated in our training ablations.

Model	WER (%) $\downarrow$		Emit delay (s) $\downarrow$		RTF $\downarrow$
Model	clean	other	clean	other	RTF $\downarrow$
VibeVoice-ASR-Streaming	3.54	9.45	1.68	1.65	0.17
Ours (Adapted ASR)	3.56	10.56	0.66	0.67	0.19

Table 1: Stage I ASR results on LibriSpeech test sets.

Model	WER(%) $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$	FPL(ms) $\downarrow$	RTF $\downarrow$
Human recording	2.14	0.734	3.527	–	–
VibeVoice-1.5B	3.04	0.689	3.971	–	–
Fixed-ratio	7.67	0.720	3.813	122	0.57
Time-aligned	3.54	0.712	4.005	130	0.66

Table 2: Stage I TTS results on Seed-TTS-Eval English.

Training configuration	ASR (LibriSpeech)					TTS (Seed-TTS-en)
	WER (%) $\downarrow$		Emission delay (s) $\downarrow$		RTF $\downarrow$	WER (%) $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$	FPL (ms) $\downarrow$	RTF $\downarrow$
	clean	other	clean	other	RTF $\downarrow$	WER (%) $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$	FPL (ms) $\downarrow$	RTF $\downarrow$
40k steps
SFT only	6.53	16.67	0.66	0.66	0.16	5.03	0.688	3.980	123	0.64
+ TTS teacher	6.76	16.88	0.66	0.66	0.18	4.31	0.702	3.995	125	0.71
+ TTS & ASR teachers	6.13	15.65	0.66	0.67	0.15	4.17	0.698	4.014	120	0.59

Table 3: Stage II ASR and TTS results on LibriSpeech and Seed-TTS-en, respectively, at 40k and 300k training steps.

Figure 2: Content-matched similarity across layers.

Unified models
Model	Mode		Backbone	Data (h)	ASR (LibriSpeech)		TTS (Seed-TTS-en)
	ASR	TTS		Data (h)	WER (%) $\downarrow$		WER (%) $\downarrow$	SIM $\uparrow$
	ASR	TTS		ASR / TTS	clean	other	WER (%) $\downarrow$	SIM $\uparrow$
HoliTok-Unite ( Li et al., 2026 )	O	O	0.5B	60–86k / 95k	5.48	12.65	7.20	0.55
Ming-UniAudio ( Yan et al., 2025 )	O	O	16B-A3B	400k / 800k	1.62	–	1.85	0.58
Streaming ASR models

Table 4: Comparison with unified and single-task streaming speech models on ASR (LibriSpeech) and TTS (Seed-TTS-Eval). S: streaming (ASR latency); O: offline. n/r: not reported.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Aligner	Mean	Start	End
CrisperWhisper	34.9	31.4	38.5
MMS-FA	37.1	43.4	30.8
Qwen3-FA	30.4	30.0	30.9
Three-way merge	24.4	24.9	23.9

Appendix

Table 5: Word-boundary mean absolute error (ms) on TIMIT. Mean averages start and end errors.

	Stage I			Stage II
Setting	TTS-TA	TTS-FR	ASR	Uni-Stream
Training steps	80k	80k	20k	300k
GPUs	2	4	4	4
Tokens per GPU	5,600	2,000	6,400	4,800
Warm-up	2%	2%	2%	0.22%

Appendix

Table 6: Training settings on H100 80 GB GPUs. TA/FR: time-aligned/fixed-ratio TTS. Uni-Stream: TTS-teacher-only configuration.

Metric	Uni-Stream $-$ specialist	Both teachers $-$ TTS teacher only
ASR WER	+0.13	$-$ 0.07
	[ $-$ 0.03, 0.29]	[ $-$ 0.22, 0.09]
TTS WER	$-$ 0.33	+0.19
	[ $-$ 1.68, 0.63]	[ $-$ 0.75, 1.41]
SIM	+0.002	$-$ 0.001
	[0.000, 0.005]	[ $-$ 0.003, 0.002]

Appendix

Table 7: Paired Stage II differences with 95% confidence intervals. The specialist comparison uses Uni-Stream trained with the TTS teacher alone.

Speech Processing

Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.

See all 412 papers

UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
May 29, 2026Zhaoqing Li, Haoning Xu, Jingran Su +9Text-to-Audio Generation TTS Synthesis
The Chinese University of Hong Kong · The Hong Kong Polytechnic University · City University of Hong Kong +3
TASTE-Streaming: Towards Streamable Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling
Mar 12, 2026Liang-Hsuan Tseng, Hung-yi LeeSpeech Language Models Audio Tokenization
Graduate Institute of Communication Engineering, National Taiwan University
Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization
Apr 21, 2026Andrei Andrusenko, Vladimir Bataev, Lilit Grigoryan +3Streaming ASR Automatic Speech Recognition
NVIDIA, Armenia · NVIDIA, USA

May 29, 2026J/K move · Enter open · S save

UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion

Zhaoqing Li, Haoning Xu, Jingran Su, Yaofang Liu+8

The Chinese University of Hong Kong · The Hong Kong Polytechnic University · City University of Hong Kong+3

Text-to-Audio Generation TTS Synthesis Latent Diffusion Models Audio Editing Unified Multimodal Models Speech Editing+1

Open paper AskPDF arXiv

We present UNISON, a latent diffusion framework that unifies speech generation, sound generation, and audio editing within a single model. A single model handles text-to-audio, text-to-speech, zero-shot speaker cloning, mixed speech-and-sound generation, scene-level audio editing, speech-in-scene editing, and timed temporal composition, all of which share a single set of weights. Our architecture features two core designs: (1) Layer-wise deep LLM fusion, which injects hidden states from uniformly sampled layers of a frozen MLLM into corresponding MM-DiT blocks via learned projections, providing depth-matched semantic conditioning that improves instruction following over single-layer baselines; and (2) a unified multi-task architecture where task identity is encoded solely by a channel-wise mask and source audio is provided through VAE-encoded channel concatenation. Training is stabilized by an online GPU-side multi-task data synthesis pipeline with task-homogeneous batching and a two-stage curriculum. With 621M--732M trainable parameters, UNISON achieves results competitive with or exceeding task-specialist models across evaluated domains, while being roughly

4\times

smaller than comparable unified systems.

Abstract

Speech Processing

Explore similar work

UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion

TASTE-Streaming: Towards Streamable Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling

Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization