cs.CLMar 28, 2026
SaveContextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild
Organizations: Argmax, Inc., USA · University of California, Los Angeles, USA
Abstract
The accuracy frontier of speech-to-text systems has plateaued on academic benchmarks.1 In contrast, industrial benchmarks and adoption in high-stakes domains suggest otherwise. We hypothesize that the primary difference between the two is contextual conditioning: Academic benchmarks are dominated by frequently encountered general vocabulary that is relatively easy to recognize compared with rare and context-defined custom vocabulary that has disproportionate impact on the usability of speech transcripts. Despite progress on contextual speech-to-text, there is no standardized benchmark. We introduce Contextual Earnings-22, an open dataset built upon Earnings-22, with realistic custom vocabulary contexts to foster research and reveal latent progress. We set six strong baselines for two dominant approaches: keyword prompting and keyword boosting. Experiments show both reach comparable and significantly improved accuracy when scaled from proof-of-concept to large-scale systems.
Figures & tables
Figure 1: Keyword F-Score vs Word Error Rate comparison across different systems and keyword contexts.
Figure 2: Contextual Earnings-22 creation pipeline. Manual review substantially reduced transcript artifacts in the overlapping portion of the dataset: 98.7% of the samples are free of inaudible and <unk> tags, and 29.5% of clips receive word-level corrections, including spelling fixes as well as word insertions and deletions, affecting 411 words in total.
| Original | Corrected |
|---|---|
| obviously also the restrictions that are still in place and that also affect our financial services activity. Okay. Very clear, thank you everyone. Okay. Uh, inaudible , Paolo, can you, uh, shed some light, uh, on the question that Artur | obviously also the restrictions that are still in place and that also affect our financial services activity. Okay. Very clear, thank you everyone. Okay. Uh, Rui Paulo, can you, uh, shed some light, uh, on the question that Arthur |
| Okay. Thanks, Panto . Appreciate that. Your next question comes from the line of Nico Margaroni’s of BRI Benelexa . Please ask your question | Okay. Thanks Mark . Appreciate that. Your next question comes from the line of Nico Margaronis of BRI Danareksa . Please ask your question |
| call is our CEO Damian Scokin, who is inaudible on our specific priorities. Lastly , our CSO will then discuss the third quarter’s financial return. After that we’ll all- we’ll open | call is our CEO Damian Scokin, who will give you an overview of the third quarter and on our specific priorities. Alberto López-Gaffney , our CFO will then discuss the third quarter’s financial return. After that we’ll all- we’ll open |
Table 1: Comparison of transcripts before (left column) and after (right column) manual correction.
| Validation | Test | |
| Samples | 130 | 630 |
| Total keyword instances | 248 | 1,259 |
| Unique keywords | 134 | 738 |
| Source files | 9 | 46 |
| Source audio duration (h) | 9.14 | 58.47 |
Table 2: Statistics of the Contextual Earnings-22 dataset. A 55-file subset yields 760 context-dense 15-second samples, split into validation (for hyperparameter tuning) and test (for benchmarking) sets at the source-audio level to prevent leakage.
| System | Contextual Keywords | Ground Truth | Prediction without Context | Prediction with Context |
|---|---|---|---|---|
| Argmax | Dan, Johan, Julien, Exane, Quenouille | So, um, next one online should be Julien Quenouille from Exane. Uh, Julien you should be live. Hello. Good morning Dan. Good morning Johan thanks, thanks for taking my questions. I have two. Uh, the first one a- and sorry if you | So um next one online should be Julien Dunnoir, Ray Sain. Um Julian uh should be live Hello good morning Don good morning Yoran thanks thanks for taking my questions. I have two. Uh the first one and sorry if you ’re | So um next one online should be Julien Dan Ray Exane Um Julien uh should be live Hello good morning Dan good morning Julien thanks thanks for taking my questions. I have two. Uh the first one and sorry if you ’re |
| Deepgram | Dan, Johan, Julien, Exane, Quenouille | So, um, next one online should be Julien Quenouille from Exane. Uh, Julien you should be live. Hello. Good morning Dan. Good morning Johan thanks, thanks for taking my questions. I have two. Uh, the first one a- and sorry if you | So next one on line should be Julien Dunois, DUMOULIN Rex Hain. Julien , should be live. SMITH:] Hello, good morning, Dan . Good morning, Jorgen. Thanks for taking my questions. I have two. The first one, and sorry if you | So next one on line should be Julien Dumoulin, DUMOULIN Rex Hain. Julien , should be live. SMITH:] Hello, good morning Dan , good morning Jorgen. Thanks for taking my questions. I have two. The first one, and sorry if you |
| OpenAI | Nico, Margaronis, BRI Danareksa | Thanks Mark. Appreciate that. Your next question comes from the line of Nico Margaronis of BRI Danareksa. Please ask your question | Thanks a lot appreciate that Your next question comes from the line of Meiko Margaronis of BRI Danarexa please ask your question | Thanks a lot appreciate that |
Table 3: Illustrative failure examples comparing STT outputs with and without context. In the predictions, blue denotes correctly predicted keywords (TP), and red denotes incorrectly predicted keywords (FP).
Figure 3: Precision and Recall comparison across different systems and keyword contexts. Dashed lines represent F-score iso-curves. See supplementary Tables S1 and S2 for exact numbers.