Although the deep integration of the Automatic Speech Recognition (ASR) system with Large Language Models (LLMs) has significantly improved accuracy, the deployment of such systems in low-latency streaming scenarios remains challenging. In this paper, we propose Uni-ASR, a unified framework based on LLMs that integrates both non-streaming and streaming speech recognition capabilities. We propose a joint training paradigm that enables the system to seamlessly transition between two recognition modes without any architectural modifications. Furthermore, we introduce a context-aware training paradigm and a co-designed fallback decoding strategy, which can enhance streaming recognition accuracy without introducing additional latency. The experimental results demonstrate that Uni-ASR not only achieves competitive performance within non-streaming mode, but also demonstrates strong effectiveness in streaming scenarios under diverse latency constraints.
Figures & tables
Figure 1: Overview of the Uni-ASR model architecture.
Figure 2: Different training paradigms of Uni-ASR.
Table 2: Comparison of ASR performance on open-source datasets. Results are reported in character error rate (CER) and word error rate (WER), covering Chinese and English benchmark datasets respectively. For Uni-ASR, beam search decoding with a width of 3 is employed; the symbol ‡ represents the streaming recognition result with a chunk length of 1000 ms.
Model
Chunk length
AISHELL
LibriSpeech
Speech ReaLLM
1920
-
3.0∣7.4
SpeechLLM-XL
1280
-
2.7∣6.7
MoCha-ASR
-
5.1∣5.5
-
Qwen3-ASR-1.7B
1000
-
1.95∣4.51
Uni-ASR
1000
2.15∣3.25
2.44∣5.71
Table 3: Comparison of streaming ASR performance on test sets. Results are reported in CER for AISHELL and WER for Librispeech.
Chunk length
Decoding method
AISHELL
LibriSpeech
Avg
1000
Greedy
2.46∣3.54
2.74∣6.65
3.85
Greedy †
2.14∣3.28
2.40∣5.82
3.41
Beam
2.15∣3.25
2.44∣5.71
3.39
Beam †
1.91∣3.04
2.22∣5.30
3.12
640
Greedy
2.63∣3.92
2.99∣7.32
4.22
Greedy †
2.36∣3.35
2.52∣6.37
3.65
Table 4: Results are reported ablation study of streaming performance across diffent chunk lengths and decoding methods. Specifically, “Beam” denotes the beam search decoding method with a width of 3, and the symbol † denotes the proposed latest-token fallback decoding strategy is employed.
Unification of automatic speech recognition (ASR) systems reduces development and maintenance costs, but training a single model to perform well in both offline and low-latency streaming settings remains challenging. We present a Unified ASR framework for Transducer (RNNT) training that supports both offline and streaming decoding within a single model, using chunk-limited attention with right context and dynamic chunked convolutions. To further close the gap between offline and streaming performance, we introduce an efficient Triton implementation of mode-consistency regularization for RNNT (MCR-RNNT), which encourages agreement across training modes. Experiments show that the proposed approach improves streaming accuracy at low latency while preserving offline performance and scaling to larger model sizes and training datasets. The proposed Unified ASR framework and the English model checkpoint are open-sourced.
Andrei Andrusenko, Vladimir Bataev, Lilit Grigoryan +3
Streaming automatic speech recognition (ASR) for real-time voice agents and full-duplex dialogue must provide accurate partial transcripts with low commit latency. Existing systems commonly use a fixed chunk size, look-ahead, or target delay, or encourage emissions near estimated acoustic boundaries. These approaches do not directly optimize how much additional context to use at each output position under a single-pass, hard-commit constraint. We propose X2Streaming-ASR, which decomposes streaming recognition into when to commit and what to commit. Its three-stage training procedure first establishes streaming recognition ability, then warm-starts the commit policy with automatically probed trajectories, and finally refines the policy using character-level, segment-assigned group-relative rewards for recognition accuracy and latency. Across ten Chinese and English test sets, X2Streaming-ASR attains the lowest mean commit latency relative to forced-aligned endpoints, 32--109ms on Chinese characters and 12--85ms on English words, while recognition accuracy remains comparable to existing systems.Code is available at https://github.com/X-Square-Robot/X2Streaming-ASR.
Open Thai automatic speech recognition (ASR) is dominated by offline, Whisper-based models that read the whole utterance before transcribing, ruling out low-latency uses such as live captioning and voice agents. We present a deployable system for streaming Thai ASR that lets a user steer its vocabulary at decode time, without retraining. A widely used open Thai model, trained with full context, collapses when run as a true stream; we restore streaming with a cache-aware encoder, by converting it or adapting a natively streaming one, and add a shallow-fusion layer that re-ranks candidates inside the streaming decoder with a GPU n-gram language model and phrase boosting. Across two Thai benchmarks and two model sizes, the streaming models stay usable where the full-context model fails, cutting character error rate 4.3-4.5x at a one-second look-ahead while running faster than real time. Decode-time steering then lifts keyword recall from 16.6% to 20.7% at no accuracy cost and negligible overhead; most of the gain comes from an n-gram over ordinary training transcripts, which resolves the written form of code-switched words the model hears but spells inconsistently, with phrase boosting adding targeted control over rare domain terms.
Warit Sirichotedumrong, Tanawin Samutsin, Shah Faisal Wani +2