Although the deep integration of the Automatic Speech Recognition (ASR) system with Large Language Models (LLMs) has significantly improved accuracy, the deployment of such systems in low-latency streaming scenarios remains challenging. In this paper, we propose Uni-ASR, a unified framework based on LLMs that integrates both non-streaming and streaming speech recognition capabilities. We propose a joint training paradigm that enables the system to seamlessly transition between two recognition modes without any architectural modifications. Furthermore, we introduce a context-aware training paradigm and a co-designed fallback decoding strategy, which can enhance streaming recognition accuracy without introducing additional latency. The experimental results demonstrate that Uni-ASR not only achieves competitive performance within non-streaming mode, but also demonstrates strong effectiveness in streaming scenarios under diverse latency constraints.
Figures & tables
Figure 1: Overview of the Uni-ASR model architecture.
Figure 2: Different training paradigms of Uni-ASR.
Table 2: Comparison of ASR performance on open-source datasets. Results are reported in character error rate (CER) and word error rate (WER), covering Chinese and English benchmark datasets respectively. For Uni-ASR, beam search decoding with a width of 3 is employed; the symbol ‡ represents the streaming recognition result with a chunk length of 1000 ms.
Model
Chunk length
AISHELL
LibriSpeech
Speech ReaLLM
1920
-
3.0∣7.4
SpeechLLM-XL
1280
-
2.7∣6.7
MoCha-ASR
-
5.1∣5.5
-
Qwen3-ASR-1.7B
1000
-
1.95∣4.51
Uni-ASR
1000
2.15∣3.25
2.44∣5.71
Table 3: Comparison of streaming ASR performance on test sets. Results are reported in CER for AISHELL and WER for Librispeech.
Chunk length
Decoding method
AISHELL
LibriSpeech
Avg
1000
Greedy
2.46∣3.54
2.74∣6.65
3.85
Greedy †
2.14∣3.28
2.40∣5.82
3.41
Beam
2.15∣3.25
2.44∣5.71
3.39
Beam †
1.91∣3.04
2.22∣5.30
3.12
640
Greedy
2.63∣3.92
2.99∣7.32
4.22
Greedy †
2.36∣3.35
2.52∣6.37
3.65
Table 4: Results are reported ablation study of streaming performance across diffent chunk lengths and decoding methods. Specifically, “Beam” denotes the beam search decoding method with a width of 3, and the symbol † denotes the proposed latest-token fallback decoding strategy is employed.