We propose Chunk-wise Attention Transducer (CHAT), a novel extension to RNN-T models that processes audio in fixed-size chunks while employing cross-attention within each chunk. This hybrid approach maintains RNN-T's streaming capability while introducing controlled flexibility for local alignment modeling. CHAT significantly reduces the temporal dimension that RNN-T must handle, yielding substantial efficiency improvements: up to 46.2% reduction in peak training memory, up to 1.36X faster training, and up to 1.69X faster inference. Alongside these efficiency gains, CHAT achieves consistent accuracy improvements over RNN-T across multiple languages and tasks -- up to 6.3% relative WER reduction for speech recognition and up to 18.0% BLEU improvement for speech translation. The method proves particularly effective for speech translation, where RNN-T's strict monotonic alignment hurts performance. Our results demonstrate that the CHAT model offers a practical solution for deploying more capable streaming speech models without sacrificing real-time constraints.
Figures & tables
category
CHAT over RNN-T
peak training GPU memory
46.2% reduction
training speed
1.36X faster
inference speed
1.69X faster
ASR WER
6.3% reduction
AST BLEU
18.0% increase
Table 1
English
German
Model
testclean
testother
vox
mls
RNNT
3.01 / 157
7.61 / 149
11.56 / 140
7.23 / 390
CHAT
2.82 / 93
7.45 / 90
11.51 / 86
7.01 / 238
rel. wer diff
-6.3%
-2.1%
-0.43%
-3.0%
rel. speed up
1.69X
1.66X
1.63X
1.64X
Table 1: English and German ASR accuracy and inference speed (WER% / decoding time (seconds, with batch=1))
Model
EN-DE
EN-ZH
EN-CA
RNN-T
29.44
34.01
18.95
CHAT
32.33
39.55
23.1
rel. diff
+9.8%
+16.3%
+18.0%
Table 2: Speech Translation Results (BLEU)
Figure 1: GPU Memory Usage (MB) when training RNNT and CHAT models for one mini-epoch with batch=32 (5000 selected utterances in Librispeech train) on A6000 GPU.
Figure 2: From top to bottom: 1. RNN-T frame alignments; 2. CHAT chunk-based alignments; 3. CHAT frame-based alignments. Chunk-size = 12.
model
chunk=6
chunk=12
chunk=24
chunk=36
RNN-T
26.63
29.44
29.57
30.60
CHAT
31.16
32.33
33.45
33.63
Table 3: En-DE AST BLEU with different chunk-sizes
batch
2
4
8
16
RNN-T
288
182
115
84
CHAT
221
125
77
56
Table 4: Comparison on batched inference speed (total seconds to decode the whole covost testset) with EN-DE AST.
model
clean
other
RNN-T
6346ms
5712ms
CHAT
6422ms
5779ms
Table 5: Averaged chunkwise token time-stamp in inference.