Streaming speech recognizers output a word only after the speaker has finished it, typically 90-400 ms later. For a voice agent every millisecond counts: the sooner the words are outputted, the sooner the agent can start a search or plan its reply. We present Earlybird, a 29M-parameter streaming recognizer (16.8 MB at 4 bits). Earlybird learns from audio cut off at random points, including partway through a word, and is asked to predict every word that is due by then in full, plus a guess of the next two words. Its raw token probabilities give a confidence for every word including the one still being spoken. On clean audiobook speech (LibriSpeech test-clean), the median word is predicted correctly 90 ms before the speaker finishes it, while released streaming models get it 90 to 400 ms after it ends. Earlybird is the earliest on all six test sets we time, covering audiobooks, TED talks, public speech, earnings calls, and synthetic speech. On the same audiobooks, 69% of words are predicted correctly before they end, against 3-25% for the other models. For words still being spoken, confidence is calibrated, with an expected calibration error (ECE) of 3.7 to 5.5. On the eight English Open ASR Leaderboard sets, it is more accurate than Moonshine Tiny on every set and roughly ties Whistle, a model of the same size.
Figures & tables
Figure 1: Word latency against accuracy; lower left is better. WER is the mean over the full Open ASR sets (Whistle: about 200 clips per set), with Earlybird at 4 bits. Latency is on the same 67 LibriSpeech clean utterances for every model except Parakeet, which was re-decoded every 240 ms on 150 other utterances (Earlybird: −80 ms on those). Hollow: published leaderboard WER.
Figure 2: Size against accuracy (left) and against word latency (right); lower left is better. Full-precision models are shown at their 16-bit size, Nemotron (streaming, 80 ms chunks) at its int8 size. Hollow: published leaderboard WER; filled: our runs. Latency as in Figure 1 .
Test set
Earlybird
Moonshine Tiny
Whistle
Whisper base
LibriSpeech clean (67)
−90 / 2.3
+270 / 3.9
+400 / 3.1
+90 / 2.3
LibriSpeech other (150)
−60 / 9.7
+300 / 13.2
+460 / 10.1
+140 / 8.7
TED-LIUM (150)
−68 / 5.4
+298 / 6.8
+505 / 7.3
+125 / 4.8
People’s Speech (150)
−66 / 14.5
+290 / 16.7
+477 / 14.2
+127 / 10.5
Earnings-22 (150)
−46 / 18.4
+309 / 21.6
+535 / 17.8
+142 / 13.3
Synthetic (67)
−73 / 5.6
+292 / 7.0
+424 / 7.5
+90 / 4.8
Table 1: Median word latency in ms relative to the end of the word (negative is before) / WER %, on the same utterances for every model within a set (count in parentheses). Revisions are words taken back per spoken word.
Test set
Earlybird
Moonshine Tiny
Whistle
Whisper base
LibriSpeech clean
68.8
3.4
9.1
25.4
TED-LIUM
52.1
1.0
2.6
17.6
People’s Speech
31.8
1.0
2.9
11.3
Synthetic
56.9
2.2
6.8
22.0
Table 2: Percent of reference words predicted correctly before they end (right, and not changed afterwards), counting transcript words and guesses output at confidence ≥0.5 . Utterances where all four models produced output: 50, 38, 100 and 50.
System
TED
VoxP
LS-c
LS-o
Earn
AMI
Giga
SPGI
Mean
Earlybird 4-bit (16.8 MB)
6.1
13.6
3.7
14.2
21.7
21.8
13.1
6.8
12.6
Earlybird full precision
5.6
12.9
3.4
12.0
20.0
19.4
11.7
6.1
11.4
Moonshine Tiny runtime
7.9
15.4
5.4
16.0
23.3
24.9
14.5
7.3
14.4
Whistle
7.7
10.8
4.1
11.3
17.9
24.0
14.6
7.1
12.2
Whisper tiny.en
5.9
13.7
5.2
14.0
17.5
22.2
11.8
5.3
12.0
Whisper base.en
4.9
9.8
3.8
10.2
14.4
18.5
11.6
4.0
9.7
Table 3: WER (%) on the eight English sets of the Open ASR Leaderboard, scored with its normalizer, every system on the same clips (TED-LIUM in full; about 200 clips of each other set, because Whistle runs at about real time per CPU core). Final transcripts, not streaming timing.
Condition
Earlybird
Moonshine Tiny
Whisper base
White noise, 10 dB
−70 / 8.7
+290 / 15.3
+140 / 6.5
White noise, 5 dB
−50 / 19.5
+320 / 29.1
+170 / 13.4
Chatter, 10 dB
−20 / 32.0
+350 / 41.5
+150 / 8.9
Chatter, 5 dB
+130 / 65.4
+490 / 79.0
+230 / 29.0
Table 4: LibriSpeech test-clean with added noise, 150 utterances: median word latency (ms) / WER %.
Variant (LibriSpeech clean, 67 utt.)
Median word latency
Revisions / word
WER
Earlybird (padding frames)
−90
1.29
2.3
Earlybird, no padding frames
−50
1.10
2.0
Earlybird 4-bit
−90
1.18
1.7
Table 5: Effect of padding frames and quantization on the same utterances.
Figure 3: Reliability of the confidence on words still being spoken: every transcript word on screen whose reference word has not ended, with all words before it right, at every 80 ms step, 200 utterances per set.
LibriSpeech
TED-LIUM
People’s Speech
Synthetic
Words still being spoken: ECE
4.3
3.7
5.5
3.7
right (%)
78.8
75.3
66.9
69.0
Every transcript word: ECE
4.3
4.7
1.4
4.0
First guessed word: ECE
0.8
1.2
3.7
1.1
Second guessed word: ECE
2.0
3.4
4.4
2.1
Table 6: Expected calibration error (points, 10 bins) of Earlybird’s raw confidence. No calibration step is applied.
Streaming automatic speech recognition (ASR) for real-time voice agents and full-duplex dialogue must provide accurate partial transcripts with low commit latency. Existing systems commonly use a fixed chunk size, look-ahead, or target delay, or encourage emissions near estimated acoustic boundaries. These approaches do not directly optimize how much additional context to use at each output position under a single-pass, hard-commit constraint. We propose X2Streaming-ASR, which decomposes streaming recognition into when to commit and what to commit. Its three-stage training procedure first establishes streaming recognition ability, then warm-starts the commit policy with automatically probed trajectories, and finally refines the policy using character-level, segment-assigned group-relative rewards for recognition accuracy and latency. Across ten Chinese and English test sets, X2Streaming-ASR attains the lowest mean commit latency relative to forced-aligned endpoints, 32--109ms on Chinese characters and 12--85ms on English words, while recognition accuracy remains comparable to existing systems.Code is available at https://github.com/X-Square-Robot/X2Streaming-ASR.
Most speech large language models (SLMs) still process speech in an offline manner. They only start responding after they receive the complete speech segment. However, many real-world applications require streaming speech interaction, for which the audio continuously flows in and the machine is required to answer questions based on the so far received audio chunks. For example, a user may ask the system to continuously count how many times a person's name has been mentioned while they are speaking. We observe a lack of benchmarks and evaluation protocols for real-time speech interaction. Motivated by this, we propose LibriSpeech-Real-QA, a dataset and a benchmark for real-time speech interaction. We construct different types of real-time question-answer pairs from LibriSpeech to evaluate whether speech LLMs can identify target words in streaming audio. We evaluate several popular real-time commercial models and find that they still tend to respond only after the speech turn has finished. We demonstrate that we can enable a pretrained SLM to perform this task, where we explore several modeling designs and establish a solid baseline for future research.
Zewen Zhan, Hui Lu, Xueyuan Chen +6
Tsinghua University, China · The Chinese University of Hong Kong, China · SenseTime Research, China
Recent advances in speech-aware language models have coupled strong acoustic encoders with large language models, enabling systems that move beyond transcription to produce richer outputs. Among these, word-level timestamp prediction is critical for applications such as captioning, media search, and multimodal synchronization, yet it is often handled by external alignment tools. In this work, we extend an existing speech-aware language model to predict timestamps directly alongside transcripts. We introduce a set of novel lightweight training strategies that improve alignment robustness while preserving recognition quality. Experiments across multiple datasets show that these strategies not only enhance timestamp accuracy, but also yield gains in overall ASR performance. Together, they demonstrate an efficient and unified approach to speech recognition with precise timestamp prediction.
Xulin Fan, Vishal Sunder, Samuel Thomas +3
University of Illinois Urbana-Champaign · IBM Research