We propose Chunk-wise Attention Transducer (CHAT), a novel extension to RNN-T models that processes audio in fixed-size chunks while employing cross-attention within each chunk. This hybrid approach maintains RNN-T's streaming capability while introducing controlled flexibility for local alignment modeling. CHAT significantly reduces the temporal dimension that RNN-T must handle, yielding substantial efficiency improvements: up to 46.2% reduction in peak training memory, up to 1.36X faster training, and up to 1.69X faster inference. Alongside these efficiency gains, CHAT achieves consistent accuracy improvements over RNN-T across multiple languages and tasks -- up to 6.3% relative WER reduction for speech recognition and up to 18.0% BLEU improvement for speech translation. The method proves particularly effective for speech translation, where RNN-T's strict monotonic alignment hurts performance. Our results demonstrate that the CHAT model offers a practical solution for deploying more capable streaming speech models without sacrificing real-time constraints.
Figures & tables
category
CHAT over RNN-T
peak training GPU memory
46.2% reduction
training speed
1.36X faster
inference speed
1.69X faster
ASR WER
6.3% reduction
AST BLEU
18.0% increase
Table 1
English
German
Model
testclean
testother
vox
mls
RNNT
3.01 / 157
7.61 / 149
11.56 / 140
7.23 / 390
CHAT
2.82 / 93
7.45 / 90
11.51 / 86
7.01 / 238
rel. wer diff
-6.3%
-2.1%
-0.43%
-3.0%
rel. speed up
1.69X
1.66X
1.63X
1.64X
Table 1: English and German ASR accuracy and inference speed (WER% / decoding time (seconds, with batch=1))
Model
EN-DE
EN-ZH
EN-CA
RNN-T
29.44
34.01
18.95
CHAT
32.33
39.55
23.1
rel. diff
+9.8%
+16.3%
+18.0%
Table 2: Speech Translation Results (BLEU)
Figure 1: GPU Memory Usage (MB) when training RNNT and CHAT models for one mini-epoch with batch=32 (5000 selected utterances in Librispeech train) on A6000 GPU.
Figure 2: From top to bottom: 1. RNN-T frame alignments; 2. CHAT chunk-based alignments; 3. CHAT frame-based alignments. Chunk-size = 12.
model
chunk=6
chunk=12
chunk=24
chunk=36
RNN-T
26.63
29.44
29.57
30.60
CHAT
31.16
32.33
33.45
33.63
Table 3: En-DE AST BLEU with different chunk-sizes
batch
2
4
8
16
RNN-T
288
182
115
84
CHAT
221
125
77
56
Table 4: Comparison on batched inference speed (total seconds to decode the whole covost testset) with EN-DE AST.
model
clean
other
RNN-T
6346ms
5712ms
CHAT
6422ms
5779ms
Table 5: Averaged chunkwise token time-stamp in inference.
Unification of automatic speech recognition (ASR) systems reduces development and maintenance costs, but training a single model to perform well in both offline and low-latency streaming settings remains challenging. We present a Unified ASR framework for Transducer (RNNT) training that supports both offline and streaming decoding within a single model, using chunk-limited attention with right context and dynamic chunked convolutions. To further close the gap between offline and streaming performance, we introduce an efficient Triton implementation of mode-consistency regularization for RNNT (MCR-RNNT), which encourages agreement across training modes. Experiments show that the proposed approach improves streaming accuracy at low latency while preserving offline performance and scaling to larger model sizes and training datasets. The proposed Unified ASR framework and the English model checkpoint are open-sourced.
Andrei Andrusenko, Vladimir Bataev, Lilit Grigoryan +3
Normally, a system that translates speech into text consists of separate modules for speech recognition and text-to-text translation. Combining those tasks into a SpeechLLM promises to exploit paralinguistic information in the speech and to reduce cascaded errors. But existing SpeechLLM systems are slow since they do not work in a real streaming fashion: they wait for a complete utterance of audio before outputting a translation, or output tokens at fixed intervals, which is not suitable for real applications. This work proposes an LLM-based architecture for real streaming speech-to-text translation. The LLM learns not just to emit output tokens, but also to decide whether it has seen enough audio to do so. The system is trained using automatic alignments of the input speech and the output text. In experiments on different language pairs, the system achieves a translation quality close to the non-streaming baseline, but with a latency of only 1-2 seconds.
Speech Large Language Models (Speech LLMs) lack a principled mechanism for streaming inference: their label-synchronous generation has no acoustic-frame alignment, making real-time decoding and end-of-utterance detection difficult. We propose TRADE TRansducer-Augmented DEcoder, which augments a multimodal LLM with a transducer branch that shares the audio encoder and uses the LLM's hidden states directly as the prediction network -- coupling frame-synchronous acoustic alignment with the LLM's linguistic reasoning. Three design choices make the system accurate, streamable, and long-form capable: (1)Tightly coupled dual vocabularies -- a compact transducer vocabulary derived from the LLM vocabulary, enabling zero-cost score fusion; (2)Chunk-synchronized streaming training with gradient stopping, eliminating the train-inference mismatch at offline-equivalent memory cost; and (3)Localized Decoder Audio Attention (LDAA), a causal sliding window that caps KV-cache memory independently of utterance length. A single TRADE checkpoint supports offline and streaming decoding across a continuous range of latency operating points. TRADE achieves 6.71% average WER on the Open ASR Leaderboard, while the streaming recognition with 960ms chunk size reaches 8.40% from the same checkpoint. On long-form speech, it obtains 3.64% WER on TED-LIUM and 10.88% on Earnings-22 without external segmentation. TRADE provides sentence-end punctuation timestamps that, when combined with acoustic voice activity detection (VAD), improve end-of-utterance detection by +0.03 F_1 over acoustic VAD alone.