Extracting insights from large audio data sets using human transcription is expensive and time-consuming. Off-the-shelf Automatic Speech Recognition (ASR) systems offer an easy-to-use scalable alternative, but systematic transcription errors, especially in domain-specific settings, can bias downstream analyses and lead to unreliable conclusions. We investigate whether imperfect ASR transcripts can still be used to obtain statistically valid keyword frequency estimates by combining model outputs with a small set of human-labeled examples, following Prediction Powered Inference and Design-based Supervised Learning frameworks. We evaluate this approach on police radio communications from Baltimore and Chicago, a domain in which keyword distributions could provide valuable insights into police activity, but off-the-shelf ASR systems exhibit systematic errors on operationally important terminology. Using bootstrap-based confidence interval coverage as our evaluation metric, we find that our approach substantially increases the number of domain-specific keywords that can be analyzed with statistically valid confidence intervals, from 43.0% using ASR alone and 27.2% using only a small set of human-labeled transcriptions to 56.2% in the Baltimore dataset, with similar improvements observed in Chicago data. Our results demonstrate that post-hoc error correction can improve statistical validity of downstream analyses even for poor-quality transcriptions, offering an avenue for better leveraging off-the-shelf ASR models for large-scale speech analysis.
Figures & tables
Fig. 1: Largest differences in estimated word counts between human transcribers and Whisper-v2-large in BPD radio communications data. Negative numbers indicate more instances of the word in human transcriptions and positive numbers indicate more instances in ASR transcripts.
Fig. 2: 95% CIs created by each estimator for hand-selected police keywords to demonstrate estimator CIs attempting to capture the “true” mean.
Fig. 3: Percentage of words where chosen ASR models and estimators created valid 95% CIs for Baltimore Police Department radio communications data.
Fig. 4: Percentage of words where chosen ASR models and estimators created valid 95% CIs for Chicago Police Department radio communications data.
Fig. 5: Relative Root Mean Squared Error (RMSE) and CI width of BPD radio communications data from θ^wHuman and θ^wMachine estimate CIs as a percentage of θ^wPP confidence RMSE. Values above 100% indicate farther estimates from the mean and wider CIs than θ^wPP .
Fig. 6: This plot shows the relationship between percentage of transcribed data and the average coverage values for human and PPI estimators over police keywords. Machine results are omitted, as that estimator assumes no labeled data exists.
Fig. 7: The relationship between word count and estimator effectiveness. Solid lines represent percentage of bootstraps in which an estimator’s successfully captured the true mean, showing how θ^wPP outperforms other estimators for rare and frequent words. Points represent the number of words with that count, and show how sparsely used the majority of the words are.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Fig. 8: Largest differences in estimated word counts between human transcribers and a finetuned Whisper-v2-large in BPD radio communications data. Negative numbers indicate more instances of the word in human transcriptions and positive numbers indicate more instances in ASR transcripts.
Pretrained ASR systems perform poorly on noisy Broadcast Police Communication (BPC), hindering efforts to understand police decision-making. Pseudo-labeling offers an unsupervised path to improve ASR without expensive human labels, but the efficacy of this approach on very noisy domains is not known. In this work, we systematically assess the opportunities and limits of pseudo-labeling to adapt foundation ASR models (Whisper and Qwen3-ASR) to noisy BPC domain corpora from Baltimore and Chicago. We demonstrate that existing internal confidence metrics (log-probabilities and STAR scores) fail to distinguish between high and low quality BPC pseudo-labels, and we introduce an external LLM-as-a-judge filtering paradigm that leverages parametric knowledge to discard contextually implausible transcripts. Our LLM-judging filters more aggressively than internal metrics and significantly reduces WER of the pseudo-labeled training sets across the Baltimore and Chicago BPC corpora, though a substantial gap remains relative to an oracle filter. We also introduce a new cross-model pseudo-labeling paradigm where one model is finetuned with pseudo-labels from the other, and we identify this method as a promising direction for future pseudo-labeling work.
Kaavya Chaparala, Su Huang, Stephen L. Morgan +2
Johns Hopkins University, Baltimore, USA · Providence College, Providence, USA
This paper analyses the implementation of Automatic Speech Recognition (ASR) into the transcription workflow of the KIParla corpus, a resource of spoken Italian. Through a two-phase experiment, 11 expert and novice transcribers produced both manual and ASR-assisted transcriptions of identical audio segments across three different types of conversation, which were subsequently analyzed through a combination of statistical modeling, word-level alignment and a series of annotation-based metrics. Results show that ASR-assisted workflows can increase transcription speed but do not consistently improve overall accuracy, with effects depending on multiple factors such as workflow configuration, conversation type and annotator experience. Analyses combining alignment-based metrics, descriptive statistics and statistical modeling provide a systematic framework to monitor transcription behavior across annotators and workflows. Despite limitations, ASR-assisted transcription, potentially supported by task-specific fine-tuning, could be integrated into the KIParla transcription workflow to accelerate corpus creation without compromising transcription quality.
Martina Simonotti, Ludovica Pannitto, Eleonora Zucchini +2
University of Bologna, Bologna - Italy · Masaryk University, Brno - Czech Republic
Automatic speech recognition systems often produce confident yet incorrect transcriptions under noisy or ambiguous conditions, which can be misleading for both users and downstream applications. Standard evaluation based on Word Error Rate focuses solely on accuracy and fails to capture transcription reliability. We introduce an abstention-aware transcription framework that enables ASR models to explicitly abstain from uncertain segments. To evaluate reliability under abstention, we propose RAS, a reliability-oriented metric that balances transcription informativeness and error aversion, with its trade-off parameter calibrated by human preference. We then train an abstention-aware ASR model through supervised bootstrapping followed by reinforcement learning. Our experiments demonstrate substantial improvements in transcription reliability while maintaining competitive accuracy.
Wenbin Huang, Yuhang Qiu, Bohan Li +5
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China · MoE Key Lab of Artificial Intelligence; Jiangsu Key Lab of Language Computing, China