SENS-ASR: Semantic Embedding injection in Neural-transducer for Streaming Automatic Speech Recognition
Authors: Youness Dkhissi, Valentin Vielzeuf, Elys Allesiardo, Anthony Larcher
Organizations: LIUM · Orange Innovation, 4 Rue du Clos Courtel, 35510 Cesson-Sévigné, France · LIUM, Le Mans Université Avenue Olivier Messiaen,72085 Le Mans CEDEX 9, France
Many Automatic Speech Recognition (ASR) applications require streaming processing of the audio data. In streaming mode, ASR systems need to start transcribing the input stream before it is complete, i.e., the systems have to process a stream of inputs with a limited (or no) future context. Compared to offline mode, this reduction of the future context degrades the performance of Streaming-ASR systems, especially while working with low-latency constraint. In this work, we present SENS-ASR, an approach to enhance the transcription quality of Streaming-ASR by reinforcing the acoustic information with semantic information. This semantic information is extracted from the available past frame-embeddings by a context module. This module is trained using knowledge distillation from a sentence embedding Language Model fine-tuned on the training dataset transcriptions. Experiments on standard datasets show that SENS-ASR significantly improves the Word Error Rate on small-chunk streaming scenarios.
Figures & tables
Figure 1: Architecture of the SENS-ASR system using an RNN-T model and a context module. Components in the red dashed-rectangle are only used during training. Components in dashed-circles are the parts of the system global loss.
Figure 2: Example of the proposed LLM paraphrasing process. The text in bold refers to the input and the output, while the strikethrough text represents the rejected paraphrases in each pruning step.
Dataset
Model
Chunk size
160ms
320ms
640ms
1,280ms
Full-context
LibriSpeech test-clean
Baseline
7.55 [7.24;7.87]
4.82 [4.57;5.06]
3.90 [3.66;4.12]
3.49 [3.27;3.69]
2.90 [2.71;3.11]
SENS-ASR
7.21 (-0.34) [6.89;7.53]
4.73 (-0.09) [4.48;4.99]
3.83 (-0.07) [3.61;4.03]
3.44 (-0.05) [3.24;3.65]
2.93(+0.03) [2.73;3.14]
LibriSpeech test-other
Baseline
18.34 [17.80;18.88]
12.41 [11.98;12.87]
9.70 [9.32;10.10]
8.39 [8.03;8.78]
6.76 [6.44;7.07]
SENS-ASR
17.89 (-0.45) [17.38;18.47]
12.11 (-0.30) [11.69;12.60]
9.66 (-0.04) [9.30;10.07]
8.55(+0.16) [8.21;8.94]
6.90(+0.14) [6.56;7.22]
TEDLIUM-2
Baseline
16.52 [15.83;17.14]
11.94 [11.39;12.52]
10.04 [9.50;10.55]
9.00 [8.52;9.51]
8.33 [7.88;8.84]
Table 1: the Word Error Rate (WER) measurements on the LibriSpeech test-clean, test-other and TEDLIUM-2 datasets with different inference chunk sizes using a unique model trained on Dynamic Chunk Training (DCT) interval of [160ms;1,280ms].The values in brackets correspond to the difference compared to the baseline and the values in square brackets represent the confidence interval of the result
Model
Additional characteristics
Chunk size(ms)
WER (%)
trimtail Song et al. (2023a)
-
640
4.68
ZeroPrompt Song et al. (2023b)
-
640
4.41
CA transformer Li et al. (2023)
Beam size=10
1,280
3.8
Delay penalized transducer Kang et al. (2023)
λ=0.006
640
3.74
Streamable decoder-only Tsunoo et al. (2024)
Beam size=10 w LM
1,600
3.2
SENS-ASR(ours)
λ=0.006
640*
3.83
Table 2: Comparison between State Of The Art models and SENS-ASR in terms of WER on LibriSpeech test-clean. For SENS-ASR, the chunk sizes with (*) were not the only targeted sizes during training stage.
Baseline
SENS-ASR
WER (%)
7.55
7.21
Number of Insertions
507
403 (-20.51%)
Number of Deletions
374
370 (-1.07%)
Number of Substitutions
3091
3020 (-2.30%)
Table 3: Error comparison between SENS-ASR and the baseline using the chunk size of 160 ms by using WER and by each type of error. The values in brackets written in bold correspond to the relative difference compared to the baseline.
Unification of automatic speech recognition (ASR) systems reduces development and maintenance costs, but training a single model to perform well in both offline and low-latency streaming settings remains challenging. We present a Unified ASR framework for Transducer (RNNT) training that supports both offline and streaming decoding within a single model, using chunk-limited attention with right context and dynamic chunked convolutions. To further close the gap between offline and streaming performance, we introduce an efficient Triton implementation of mode-consistency regularization for RNNT (MCR-RNNT), which encourages agreement across training modes. Experiments show that the proposed approach improves streaming accuracy at low latency while preserving offline performance and scaling to larger model sizes and training datasets. The proposed Unified ASR framework and the English model checkpoint are open-sourced.
Andrei Andrusenko, Vladimir Bataev, Lilit Grigoryan +3
Although the deep integration of the Automatic Speech Recognition (ASR) system with Large Language Models (LLMs) has significantly improved accuracy, the deployment of such systems in low-latency streaming scenarios remains challenging. In this paper, we propose Uni-ASR, a unified framework based on LLMs that integrates both non-streaming and streaming speech recognition capabilities. We propose a joint training paradigm that enables the system to seamlessly transition between two recognition modes without any architectural modifications. Furthermore, we introduce a context-aware training paradigm and a co-designed fallback decoding strategy, which can enhance streaming recognition accuracy without introducing additional latency. The experimental results demonstrate that Uni-ASR not only achieves competitive performance within non-streaming mode, but also demonstrates strong effectiveness in streaming scenarios under diverse latency constraints.
Yinfeng Xia, Jian Tang, Junfeng Hou +2
Qwen Applications Business Group, Alibaba, China · Tongyi AI Lab, Alibaba, China
Streaming automatic speech recognition (ASR) for real-time voice agents and full-duplex dialogue must provide accurate partial transcripts with low commit latency. Existing systems commonly use a fixed chunk size, look-ahead, or target delay, or encourage emissions near estimated acoustic boundaries. These approaches do not directly optimize how much additional context to use at each output position under a single-pass, hard-commit constraint. We propose X2Streaming-ASR, which decomposes streaming recognition into when to commit and what to commit. Its three-stage training procedure first establishes streaming recognition ability, then warm-starts the commit policy with automatically probed trajectories, and finally refines the policy using character-level, segment-assigned group-relative rewards for recognition accuracy and latency. Across ten Chinese and English test sets, X2Streaming-ASR attains the lowest mean commit latency relative to forced-aligned endpoints, 32--109ms on Chinese characters and 12--85ms on English words, while recognition accuracy remains comparable to existing systems.Code is available at https://github.com/X-Square-Robot/X2Streaming-ASR.