Modern voice-agent systems rely on streaming speech recognition models that operate under stringent latency constraints. This study shows that, due to the limited memory constraints of real-time processing, these systems are adversely impacted by conversational phenomena such as long silences and backchannels. While many agentic pipelines mitigate this by resetting state at each turn, this approach discards vital context and impairs performance at turn onsets. We propose two state-management strategies that preserve cross-utterance context to reduce onset errors. In experiments with two state-of-the-art streaming models on two spoken dialogue benchmarks, our best method yields an average of 15-21% relative WER reduction at utterance onsets.
Figures & tables
Figure 1: Stateful inference. Cached acoustic memory state for streaming ASR models under the different state-management strategies defined in Section 3.3 .
Figure 2: Word Error Rates for different state-management strategies with fastconformer-114m on Switchboard.
Model
Category
CallHome
Switchboard
fs
cs
sc
rb
fs
cs
sc
rb
fastconformer-114m
Overall
15.43
15.18
14.02
13.83
11.97
12.36
10.26
10.20
First Word
48.10
41.40
36.24
35.03
51.44
44.32
35.92
34.78
After Long Pause
17.17
20.66
16.15
15.70
14.73
20.06
13.08
12.99
After Short Turn
18.52
20.20
17.17
16.61
13.93
15.94
12.03
11.94
nemotron-streaming-0.6b
Overall
13.82
14.29
13.86
13.52
10.90
11.40
10.85
10.62
Table 1: Detailed WER (%) analysis. First Word refers to the WER of the initial word in a turn; “Long Pause” denotes silences exceeding the acoustic memory length of 5.6 s; “Short Turn” denotes turns shorter than 3 words.
Figure 3: Comparison of SC and RB for Switchboard.
fastconformer-114m
nemotron-streaming-0.6b
Shape
Memory
Shape
Memory
Encoder State
Attention KV Cache
(17,1,70,512)
2.32 MB
(24,1,70,1024)
6.56 MB
Convolution Cache
(17,1,512,8)
0.27 MB
(24,1,1024,8)
0.75 MB
Valid Cache Length
(1,)
8 B
(1,)
8 B
Decoder State
LSTM Recurrent State
(3,640)
7.5 KB
(10,640)
25 KB
Last Token
scalar
8 B
scalar
8 B
Table 2: Model state components with the corresponding tensor shapes and memory footprint.
Dataset
Model
Δ % (FS → RB)
MAPSSWE
Wilcoxon
Switchboard
nemotron-streaming-0.6b
2.47
∗∗
∗∗
fastconformer-114m
14.36
∗∗
∗∗
CallHome
nemotron-streaming-0.6b
2.17
∗
∗∗
fastconformer-114m
9.91
∗∗
∗∗
Table 3: Relative WER reduction (%) from FS to RB; statistical significance is assessed using the MAPSSWE and per-utterance Wilcoxon signed-rank tests. ∗ indicates p<0.05 and ∗∗ indicates p<0.001 .
Model
Dataset
SC
RB ( k=1 )
RB ( k=5 )
RB ( k=10 )
fastconformer-114m
CallHome (val)
13.94
13.82
13.70
13.74
Switchboard (val)
9.95
9.90
9.87
9.91
CallHome (test)
14.02
13.92
13.83
13.90
Switchboard (test)
10.26
10.21
10.20
10.25
nemotron-streaming-0.6b
CallHome (val)
13.83
13.75
13.66
13.57
Switchboard (val)
10.37
10.30
10.05
10.02
Table 4: Comparison of overall WER (%) across different token thresholds for RB.
Model
Latency
CallHome
Switchboard
fs
cs
sc
rb
fs
cs
sc
rb
fastconformer-114m
80 ms
15.52
15.64
14.47
14.19
12.21
12.81
10.57
10.49
160 ms
15.43
15.18
14.02
13.83
11.97
12.36
10.26
10.20
240 ms
15.45
15.37
14.06
13.98
12.04
12.66
10.28
10.27
560 ms
14.75
15.04
13.49
13.44
11.44
12.06
10.04
10.04
nemotron-streaming-0.6b
80 ms
14.51
14.86
14.35
14.16
11.44
11.88
11.5
11.17
Table 5: Comparison of overall WER (%) across different state-management strategies and latency settings.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Model
Category
FS
CS
SC
RB
fastconformer-114m
Overall
14.71
15.98
13.84
13.59
First Word
49.15
45.92
35.05
32.50
After Long Pause
18.17
23.33
17.13
16.97
After Short Turn
16.61
19.41
15.63
15.39
nemotron-streaming-0.6b
Overall
14.79
15.74
14.66
14.28
First Word
33.15
37.72
32.95
30.72
Appendix
Table 6: Performance comparison of different state-management strategies on Switchboard with 160 ms latency when turn boundaries are derived from Silero-VAD. Metric definitions are identical to those used in Table 1 .