cs.CLMar 28, 2026

Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild

Authors: Berkin Durmus, Chen Cen, Eduardo Pacheco, Arda Okan, Atila Orhon

Organizations: Argmax, Inc., USA · University of California, Los Angeles, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The accuracy frontier of speech-to-text systems has plateaued on academic benchmarks.1 In contrast, industrial benchmarks and adoption in high-stakes domains suggest otherwise. We hypothesize that the primary difference between the two is contextual conditioning: Academic benchmarks are dominated by frequently encountered general vocabulary that is relatively easy to recognize compared with rare and context-defined custom vocabulary that has disproportionate impact on the usability of speech transcripts. Despite progress on contextual speech-to-text, there is no standardized benchmark. We introduce Contextual Earnings-22, an open dataset built upon Earnings-22, with realistic custom vocabulary contexts to foster research and reveal latent progress. We set six strong baselines for two dominant approaches: keyword prompting and keyword boosting. Experiments show both reach comparable and significantly improved accuracy when scaled from proof-of-concept to large-scale systems.

Figures & tables

Figure 1: Keyword F-Score vs Word Error Rate comparison across different systems and keyword contexts.
Figure 2: Contextual Earnings-22 creation pipeline. Manual review substantially reduced transcript artifacts in the overlapping portion of the dataset: 98.7% of the samples are free of inaudible and <unk> tags, and 29.5% of clips receive word-level corrections, including spelling fixes as well as word insertions and deletions, affecting 411 words in total.
OriginalCorrected
obviously also the restrictions that are still in place and that also affect our financial services activity. Okay. Very clear, thank you everyone. Okay. Uh, inaudible , Paolo, can you, uh, shed some light, uh, on the question that Arturobviously also the restrictions that are still in place and that also affect our financial services activity. Okay. Very clear, thank you everyone. Okay. Uh, Rui Paulo, can you, uh, shed some light, uh, on the question that Arthur
  
Okay. Thanks, Panto . Appreciate that. Your next question comes from the line of Nico Margaroni’s of BRI Benelexa . Please ask your questionOkay. Thanks Mark . Appreciate that. Your next question comes from the line of Nico Margaronis of BRI Danareksa . Please ask your question
  
call is our CEO Damian Scokin, who is inaudible on our specific priorities. Lastly , our CSO will then discuss the third quarter’s financial return. After that we’ll all- we’ll opencall is our CEO Damian Scokin, who will give you an overview of the third quarter and on our specific priorities. Alberto López-Gaffney , our CFO will then discuss the third quarter’s financial return. After that we’ll all- we’ll open
  
Table 1: Comparison of transcripts before (left column) and after (right column) manual correction.
 ValidationTest
Samples130630
Total keyword instances2481,259
Unique keywords134738
Source files946
Source audio duration (h)9.1458.47
Table 2: Statistics of the Contextual Earnings-22 dataset. A 55-file subset yields 760 context-dense 15-second samples, split into validation (for hyperparameter tuning) and test (for benchmarking) sets at the source-audio level to prevent leakage.
SystemContextual KeywordsGround TruthPrediction without ContextPrediction with Context
ArgmaxDan, Johan, Julien, Exane, QuenouilleSo, um, next one online should be Julien Quenouille from Exane. Uh, Julien you should be live. Hello. Good morning Dan. Good morning Johan thanks, thanks for taking my questions. I have two. Uh, the first one a- and sorry if youSo um next one online should be Julien Dunnoir, Ray Sain. Um Julian uh should be live Hello good morning Don good morning Yoran thanks thanks for taking my questions. I have two. Uh the first one and sorry if you ’reSo um next one online should be Julien Dan Ray Exane Um Julien uh should be live Hello good morning Dan good morning Julien thanks thanks for taking my questions. I have two. Uh the first one and sorry if you ’re
     
DeepgramDan, Johan, Julien, Exane, QuenouilleSo, um, next one online should be Julien Quenouille from Exane. Uh, Julien you should be live. Hello. Good morning Dan. Good morning Johan thanks, thanks for taking my questions. I have two. Uh, the first one a- and sorry if youSo next one on line should be Julien Dunois, DUMOULIN Rex Hain. Julien , should be live. SMITH:] Hello, good morning, Dan . Good morning, Jorgen. Thanks for taking my questions. I have two. The first one, and sorry if youSo next one on line should be Julien Dumoulin, DUMOULIN Rex Hain. Julien , should be live. SMITH:] Hello, good morning Dan , good morning Jorgen. Thanks for taking my questions. I have two. The first one, and sorry if you
     
OpenAINico, Margaronis, BRI DanareksaThanks Mark. Appreciate that. Your next question comes from the line of Nico Margaronis of BRI Danareksa. Please ask your questionThanks a lot appreciate that Your next question comes from the line of Meiko Margaronis of BRI Danarexa please ask your questionThanks a lot appreciate that
     
Table 3: Illustrative failure examples comparing STT outputs with and without context. In the predictions, blue denotes correctly predicted keywords (TP), and red denotes incorrectly predicted keywords (FP).
Figure 3: Precision and Recall comparison across different systems and keyword contexts. Dashed lines represent F-score iso-curves. See supplementary Tables S1 and S2 for exact numbers.
More in this topic

Contextual ASR

See all 19 papers

Explore similar work

CardsList
Apr 12, 2026

Knowing What to Stress: A Discourse-Conditioned Text-to-Speech Benchmark

Spoken meaning often depends not only on what is said, but also on which word is emphasized. The same sentence can convey correction, contrast, or clarification depending on where emphasis falls. Although modern text-to-speech (TTS) systems generate expressive speech, it remains unclear whether they infer contextually appropriate stress from discourse alone. To address this gap, we present Context-Aware Stress TTS (CAST), a benchmark for evaluating context-conditioned word-level stress in TTS. Items are defined as contrastive context pairs: identical sentences paired with distinct contexts requiring different stressed words. We evaluate state-of-the-art systems and find a consistent gap: text-only language models reliably recover the intended stress from context, yet TTS systems frequently fail to realize it in speech. We release the benchmark, evaluation framework, construction pipeline and a synthetic corpus to support future work on context-aware speech synthesis.

Arnon Turetzky, Avihu Dekel, Hagai Aronowitz +2

The Hebrew University of Jerusalem, Israel · IBM Research, Israel

TTS SynthesisTTS EvaluationSpeech Prosody
AskPDF
Jul 26, 2026

Earnings25: A Comprehensive 500-Hour Speech Benchmark for Finance

We introduce Earnings25, a finance-domain benchmark for evaluating automatic speech recognition (ASR) on English-language earnings calls under realistic conditions. Earnings25 comprises two complementary test sets: (i) testset-full, 498 hours of full English-language S&P 500 earnings calls from Q4 2025, and (ii) testset-segmented, a 46-hour industry-balanced set of 290 segments sampled from English-language U.S. earnings calls in 2025. The benchmark provides aligned transcripts and structured metadata, including speaker roles, industry labels, and call structure, enabling speaker- and industry-aware evaluation beyond aggregate word error rate (WER). We report reproducible baselines for Whisper and Parakeet-TDT using standardized scoring.

Denglin Jiang, Haoran Zhou, Anshul Wadhawan +7

Bloomberg, United States

ASR EvaluationSpeech Recognition BenchmarksAutomatic Speech RecognitionFinance
AskPDF
Aug 6, 2026

How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMs

Recognizing new and rare words - named entities, acronyms, domain specific special words, and other items scarce in training data - remains a key challenge for automatic speech recognition (ASR). We compare two strategies for this: context biasing methods, where an ASR model is extended such that during inference a word list can be supplied, and speech large language models (LLMs) prompted with context directly. We evaluate two context biasing methods based on Whisper against three speech LLMs across read and non-read speech, reporting biased, unbiased, and overall word error rate (WER). The context biasing methods cut biased WER by up to 88% relative while leaving other words largely unaffected. Speech LLMs excel on read speech but generalize less well to non-read speech, and prove sensitive to distractor count and prompt word order. We characterize the resulting trade-offs to guide method selection.

Christian Huber, Alexander Waibel

Interactive Systems Lab · Karlsruhe Institute of Technology · Karlsruhe, Germany +2

ASR EvaluationAutomatic Speech RecognitionSpeech Language ModelsContextual ASR
AskPDF