SENS-ASR: Semantic Embedding injection in Neural-transducer for Streaming Automatic Speech Recognition
Authors: Youness Dkhissi, Valentin Vielzeuf, Elys Allesiardo, Anthony Larcher
Organizations: LIUM · Orange Innovation, 4 Rue du Clos Courtel, 35510 Cesson-Sévigné, France · LIUM, Le Mans Université Avenue Olivier Messiaen,72085 Le Mans CEDEX 9, France
Many Automatic Speech Recognition (ASR) applications require streaming processing of the audio data. In streaming mode, ASR systems need to start transcribing the input stream before it is complete, i.e., the systems have to process a stream of inputs with a limited (or no) future context. Compared to offline mode, this reduction of the future context degrades the performance of Streaming-ASR systems, especially while working with low-latency constraint. In this work, we present SENS-ASR, an approach to enhance the transcription quality of Streaming-ASR by reinforcing the acoustic information with semantic information. This semantic information is extracted from the available past frame-embeddings by a context module. This module is trained using knowledge distillation from a sentence embedding Language Model fine-tuned on the training dataset transcriptions. Experiments on standard datasets show that SENS-ASR significantly improves the Word Error Rate on small-chunk streaming scenarios.
Figures & tables
Figure 1: Architecture of the SENS-ASR system using an RNN-T model and a context module. Components in the red dashed-rectangle are only used during training. Components in dashed-circles are the parts of the system global loss.
Figure 2: Example of the proposed LLM paraphrasing process. The text in bold refers to the input and the output, while the strikethrough text represents the rejected paraphrases in each pruning step.
Dataset
Model
Chunk size
160ms
320ms
640ms
1,280ms
Full-context
LibriSpeech test-clean
Baseline
7.55 [7.24;7.87]
4.82 [4.57;5.06]
3.90 [3.66;4.12]
3.49 [3.27;3.69]
2.90 [2.71;3.11]
SENS-ASR
7.21 (-0.34) [6.89;7.53]
4.73 (-0.09) [4.48;4.99]
3.83 (-0.07) [3.61;4.03]
3.44 (-0.05) [3.24;3.65]
2.93(+0.03) [2.73;3.14]
LibriSpeech test-other
Baseline
18.34 [17.80;18.88]
12.41 [11.98;12.87]
9.70 [9.32;10.10]
8.39 [8.03;8.78]
6.76 [6.44;7.07]
SENS-ASR
17.89 (-0.45) [17.38;18.47]
12.11 (-0.30) [11.69;12.60]
9.66 (-0.04) [9.30;10.07]
8.55(+0.16) [8.21;8.94]
6.90(+0.14) [6.56;7.22]
TEDLIUM-2
Baseline
16.52 [15.83;17.14]
11.94 [11.39;12.52]
10.04 [9.50;10.55]
9.00 [8.52;9.51]
8.33 [7.88;8.84]
Table 1: the Word Error Rate (WER) measurements on the LibriSpeech test-clean, test-other and TEDLIUM-2 datasets with different inference chunk sizes using a unique model trained on Dynamic Chunk Training (DCT) interval of [160ms;1,280ms].The values in brackets correspond to the difference compared to the baseline and the values in square brackets represent the confidence interval of the result
Model
Additional characteristics
Chunk size(ms)
WER (%)
trimtail Song et al. (2023a)
-
640
4.68
ZeroPrompt Song et al. (2023b)
-
640
4.41
CA transformer Li et al. (2023)
Beam size=10
1,280
3.8
Delay penalized transducer Kang et al. (2023)
λ=0.006
640
3.74
Streamable decoder-only Tsunoo et al. (2024)
Beam size=10 w LM
1,600
3.2
SENS-ASR(ours)
λ=0.006
640*
3.83
Table 2: Comparison between State Of The Art models and SENS-ASR in terms of WER on LibriSpeech test-clean. For SENS-ASR, the chunk sizes with (*) were not the only targeted sizes during training stage.
Baseline
SENS-ASR
WER (%)
7.55
7.21
Number of Insertions
507
403 (-20.51%)
Number of Deletions
374
370 (-1.07%)
Number of Substitutions
3091
3020 (-2.30%)
Table 3: Error comparison between SENS-ASR and the baseline using the chunk size of 160 ms by using WER and by each type of error. The values in brackets written in bold correspond to the relative difference compared to the baseline.