LibriSpeech-Real-QA: A Benchmark for Streaming Question Answering
Organizations: Tsinghua University, China · The Chinese University of Hong Kong, China · SenseTime Research, China
Abstract
Most speech large language models (SLMs) still process speech in an offline manner. They only start responding after they receive the complete speech segment. However, many real-world applications require streaming speech interaction, for which the audio continuously flows in and the machine is required to answer questions based on the so far received audio chunks. For example, a user may ask the system to continuously count how many times a person's name has been mentioned while they are speaking. We observe a lack of benchmarks and evaluation protocols for real-time speech interaction. Motivated by this, we propose LibriSpeech-Real-QA, a dataset and a benchmark for real-time speech interaction. We construct different types of real-time question-answer pairs from LibriSpeech to evaluate whether speech LLMs can identify target words in streaming audio. We evaluate several popular real-time commercial models and find that they still tend to respond only after the speech turn has finished. We demonstrate that we can enable a pretrained SLM to perform this task, where we explore several modeling designs and establish a solid baseline for future research.
Figures & tables
| Set | Split | Lex. det. | Lex. cnt. | Sem. det. | Sem. cnt. |
| Train | train-other-500 | 138,800 | 38,282 | 199,111 | 38,889 |
| train-clean-360 | 92,593 | 28,804 | 137,485 | 27,584 | |
| train-clean-100 | 26,529 | 7,639 | 40,260 | 8,014 | |
| Dev | dev-clean | 2,511 | 243 | 2,437 | 309 |
| dev-other | 2,623 | 239 | 2,300 | 260 | |
| Benchmark | test-clean | 2,399 | 237 | 2,392 | 277 |
| Model | Win. | F1 | Recall | Precision | |
|---|---|---|---|---|---|
| Gemini Live 3.1 Flash | – | 0.83 / 0.87 | 0.98 / 0.96 | 0.72 / 0.79 | 5.00 / 4.35 |
| GPT-realtime-2.1-mini | – | 0.94 / 0.83 | 0.98 / 0.88 | 0.91 / 0.79 | 5.43 / 6.52 |
| Qwen-Audio Flash | – | 0.79 / 0.81 | 0.78 / 0.81 | 0.80 / 0.81 | 4.85 / 5.22 |
| Qwen-Audio Plus | – | 0.78 / 0.79 | 0.78 / 0.79 | 0.79 / 0.78 | 4.84 / 5.47 |
| Qwen2.5-Omni-7B † | – | 0.50 / 0.51 | 0.50 / 0.50 | 0.51 / 0.52 | 1.457 / 1.328 |
| Offline † | – | 0.94 / 0.91 | 0.93 / 0.91 | 0.94 / 0.92 | 0.010 / 0.003 |
| Model | Win. | F1 | Recall | Precision | |
|---|---|---|---|---|---|
| Gemini Live 3.1 Flash | – | 0.61 / 0.68 | 0.68 / 0.68 | 0.56 / 0.69 | 8.94 / 7.15 |
| GPT-realtime-2.1-mini | – | 0.58 / 0.46 | 0.45 / 0.36 | 0.85 / 0.61 | 10.07 / 10.03 |
| Qwen-Audio Flash | – | 0.44 / 0.42 | 0.32 / 0.34 | 0.67 / 0.54 | 4.03 / 6.52 |
| Qwen-Audio Plus | – | 0.44 / 0.45 | 0.32 / 0.35 | 0.68 / 0.64 | 4.07 / 7.03 |
| Qwen2.5-Omni-7B † | – | 0.07 / 0.08 | 0.10 / 0.13 | 0.05 / 0.06 | 2.419 / 2.990 |
| Offline † | – | 0.98 / 0.96 | 0.97 / 0.97 | 0.99 / 0.96 | 0.062 / 0.023 |
| Model | Win. | F1 | Recall | Precision | |
|---|---|---|---|---|---|
| Gemini Live 3.1 Flash | – | 0.65 / 0.61 | 0.70 / 0.65 | 0.61 / 0.57 | 5.89 / 5.09 |
| GPT-realtime-2.1-mini | – | 0.73 / 0.44 | 0.72 / 0.45 | 0.73 / 0.44 | 6.64 / 6.66 |
| Qwen-Audio Flash | – | 0.52 / 0.45 | 0.49 / 0.44 | 0.56 / 0.46 | 3.25 / 6.13 |
| Qwen-Audio Plus | – | 0.54 / 0.47 | 0.51 / 0.46 | 0.57 / 0.50 | 3.23 / 6.32 |
| Qwen2.5-Omni-7B † | – | 0.25 / 0.24 | 0.26 / 0.23 | 0.25 / 0.25 | 1.758 / 1.716 |
| Offline † | – | 0.78 / 0.66 | 0.75 / 0.64 | 0.81 / 0.69 | 0.021 / 0.013 |
| Model | Win. | F1 | Recall | Precision | |
|---|---|---|---|---|---|
| Gemini Live 3.1 Flash | – | 0.49 / 0.54 | 0.59 / 0.54 | 0.42 / 0.54 | 8.86 / 8.04 |
| GPT-realtime-2.1-mini | – | 0.48 / 0.45 | 0.37 / 0.35 | 0.69 / 0.65 | 10.21 / 11.27 |
| Qwen-Audio Flash | – | 0.40 / 0.43 | 0.30 / 0.35 | 0.61 / 0.55 | 4.13 / 7.20 |
| Qwen-Audio Plus | – | 0.43 / 0.48 | 0.32 / 0.37 | 0.68 / 0.69 | 4.11 / 7.60 |
| Qwen2.5-Omni-7B † | – | 0.04 / 0.02 | 0.04 / 0.01 | 0.05 / 0.03 | 3.074 / 1.748 |
| Offline † | – | 0.91 / 0.89 | 0.89 / 0.86 | 0.92 / 0.92 | 0.220 / 0.128 |
| pos:neg | Task | F1 | Recall | Precision |
|---|---|---|---|---|
| 1.0 : 0.0 | Lexical det. | 0.952 | 0.960 | 0.943 |
| Semantic det. | 0.715 | 0.751 | 0.683 | |
| Lexical cnt. | 0.926 | 0.944 | 0.909 | |
| Semantic cnt. | 0.724 | 0.780 | 0.676 | |
| 0.9 : 0.1 | Lexical det. | 0.972 | 0.979 | 0.966 |
| Semantic det. | 0.750 | 0.776 | 0.727 |
Speech Language Models
Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.
See all 327 papers