cs.CLNov 7, 2026

KeyASRBench: Measuring Whole-Phrase Keyword Preservation in Real-World ASR

Authors: Guanzheng Wei, Ziyi Zhao

Organizations: Southwest University · University of Chinese Academy of Social Sciences

Abstract

Public automatic speech recognition (ASR) benchmarks increasingly report low sequence error rates, yet production systems still use hotword lists, contextual biasing, retrieval, and post-correction. This motivates a complementary question: does a transcript preserve selected names, products, technical terms, versions, and relational phrases? We present KeyASRBench, a bilingual pilot that operationalizes occurrence-based Keyword Error Rate (KER) over frozen whole-phrase annotations. The 10-hour model-conditioned challenge set contains 1,501 Chinese and English clips with 4,723 keyword occurrences. Across five locally inferred checkpoints and two official API systems, language-specific MER/WER ranges from 5.31% to 12.91%, while KER ranges from 13.61% to 33.81%. A disjoint 30-minute model-blind control yields KER of 10.65%-27.31%. The control confirms that selected-phrase errors extend beyond challenge-set selection.

Figures & tables

Explore similar work

CardsList
  1. Conducting Reliable Keyword Estimation with Unreliable ASR in Police Audio Data

    Oct 19, 2026Robert Cole Molloy, Rhiannon N. Miller, Stephen Morgan +1Supervised LearningAutomatic Speech Recognition

  2. Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER

    Sep 18, 2026Hritika Sharma, Thibault Bañeras-Roux, Alessandra Pinto +4ASR EvaluationSemantic Textual Similarity