KeyASRBench: Measuring Whole-Phrase Keyword Preservation in Real-World ASR
Organizations: Southwest University · University of Chinese Academy of Social Sciences
Abstract
Public automatic speech recognition (ASR) benchmarks increasingly report low sequence error rates, yet production systems still use hotword lists, contextual biasing, retrieval, and post-correction. This motivates a complementary question: does a transcript preserve selected names, products, technical terms, versions, and relational phrases? We present KeyASRBench, a bilingual pilot that operationalizes occurrence-based Keyword Error Rate (KER) over frozen whole-phrase annotations. The 10-hour model-conditioned challenge set contains 1,501 Chinese and English clips with 4,723 keyword occurrences. Across five locally inferred checkpoints and two official API systems, language-specific MER/WER ranges from 5.31% to 12.91%, while KER ranges from 13.61% to 33.81%. A disjoint 30-minute model-blind control yields KER of 10.65%-27.31%. The control confirms that selected-phrase errors extend beyond challenge-set selection.
Figures & tables
| Property | Value |
|---|---|
| Total duration | 9.999 h |
| Chinese / English | 5.000 / 5.000 h |
| Segments | 1,501 (750 Chinese / 751 English) |
| Source-origin partitions A / B | 751 / 750 clips |
| Recording groups | 273 |
| Historical selection label (quota target) | 55.0% |
| Model | Chinese MER | English WER | Overall KER | Chinese KER | English KER |
|---|---|---|---|---|---|
| Qwen3-ASR-0.6B | 8.19% | 8.52% | 33.81% (29.38–38.14) | 43.81% | 22.20% |
| Qwen3-ASR-1.7B | 5.35% | 5.95% | 24.88% (21.12–28.58) | 33.49% | 14.87% |
| GLM-ASR-Nano-2512 | 8.77% | 8.71% | 32.50% (28.11–36.67) | 43.07% | 20.23% |
| Whisper large-v3-turbo | 12.91% | 8.74% | 31.29% (26.67–35.62) | 43.89% | 16.66% |
| MOSS-Transcribe-Diarize-0.9B | 8.22% | 8.65% | 32.29% (27.09–36.93) | 46.49% | 15.79% |
| StepAudio 3 ASR Max | 5.31% | 6.27% | 13.61% (10.90–16.27) | 18.87% | 7.51% |
| System | Model-blind control KER | 10-hour challenge KER | Challenge control |
|---|---|---|---|
| Qwen3-ASR-0.6B | 24.54% (17.79–31.78) | 33.81% | +9.27 pp |
| Qwen3-ASR-1.7B | 15.74% (10.30–21.49) | 24.88% | +9.14 pp |
| GLM-ASR-Nano-2512 | 21.30% (14.56–28.38) | 32.50% | +11.20 pp |
| Whisper large-v3-turbo | 27.31% (19.07–35.81) | 31.29% | +3.98 pp |
| MOSS-Transcribe-Diarize-0.9B | 25.46% (18.57–33.18) | 32.29% | +6.83 pp |
| StepAudio 3 ASR Max | 10.65% (6.79–14.75) | 13.61% | +2.96 pp |
| Case type | Ground truth | System | Recognized form(s) | Sent. MER/WER | Sent. KER |
|---|---|---|---|---|---|
| ZH-entity | 极客公园; 宇树科技 | Qwen-0.6B | 极公源 ; 宇树科技 | 6.25% MER | 50% |
| Qwen-1.7B | 极工源 ; 语数科技 | 10.42% MER | 100% | ||
| GLM-1.5B | 极光源 ; 语数科技 | 10.42% MER | 100% | ||
| MOSS-0.9B | 极光园 ; 语数科技 | 8.33% MER | 100% | ||
| StepAudio | 极公园 ; 宇树科技 | 10.42% MER | 50% | ||
| Doubao | 极公园 ; 宇树科技 | 10.42% MER | 50% |
ASR Evaluation
Automatic speech recognition (ASR) evaluation measures transcript accuracy, robustness, fairness, and usefulness across languages, speakers, and real-world conditions.
See all 151 papers