Automatic speech recognition (ASR) pipelines often normalize transcripts by removing filled pauses and punctuation and expanding colloquial forms, potentially discarding cues useful for dialogue act classification. We evaluate nine transcript conditions across six classifiers (TF-IDF logistic regression, RoBERTa, BERT, ALBERT, GPT-2, and BiLSTM) and three conversational corpora (Switchboard, ICSI MRDA, and SLUE-HVB), supplemented by multi-seed experiments, normalization controls, and real-ASR evaluations. Across five random seeds, heavy normalization reduces Macro-F1 in all 15 neural model-corpus combinations under the original preprocessing protocols. Removal using the original filler and backchannel lexicon consistently degrades performance on MRDA and SLUE-HVB but produces mixed effects on Switchboard. Punctuation-aware TF-IDF experiments show that punctuation removal reduces Macro-F1 by 0.036 on Switchboard and 0.078 on MRDA, demonstrating that the original linear baseline's apparent insensitivity resulted from tokenization. Restricting filler removal to um, uh, and erm preserves degradation on MRDA but largely eliminates the SLUE-HVB effect, where the original lexicon removes class-defining backchannel expressions. These findings demonstrate that normalization effects depend on preprocessing definitions, feature extraction, and conversational context, supporting task-dependent rather than indiscriminate transcript normalization.
Figures & tables
Switchboard
MRDA
HVB
Register
Telephone
Meetings
Call-center
Train/test
34.6k/9.1k
75.1k/16.7k
11.3k/6.1k
Labels
27
12
16
Canonical filler rate a
40.5%
15.9%
5.7%
TABLE I: Dataset statistics after rare-label filtering ( ≥50 occurrences).
Corpus
Model
Gold F1
Filler F1
Heavy F1
Switchboard
TF-IDF+LR
0.311
0.300
0.293
RoBERTa
0.449±0.006
0.447±0.005
0.393±0.012
BERT
0.440±0.005
0.449±0.008
0.387±0.004
ALBERT
0.420±0.008
0.409±0.026
0.355±0.046
GPT-2
0.388±0.012
0.383±0.009
0.339±0.008
BiLSTM
0.370±0.007
0.374±0.012
0.336±0.010
TABLE II: Macro-F1 under original, filler-removed, and heavily normalized transcripts. Neural results are mean ± standard deviation over five seeds. TF-IDF+LR is deterministic. Original corpus-specific normalization protocols are used.
Corpus
Default Δ
Aware Gold
Aware Δ
Switchboard
0.0002
0.360
0.036
MRDA
0.0000
0.494
0.078
SLUE-HVB
0.0000
0.717
0.000
TABLE III: Punctuation-removal effects for TF-IDF+LR. Δpunct denotes the Macro-F1 decrease relative to the corresponding gold condition.
Model
SWB
MRDA
HVB
TF-IDF+LR
0.007
0.039
0.015
RoBERTa
0.044
0.080
−0.004
BERT
0.041
0.076
−0.001
ALBERT
0.032
0.077
0.003
GPT-2
0.048
0.110
−0.049
BiLSTM
0.022
0.056
0.000
TABLE IV: Primary-seed Macro-F1 degradation under harmonized strict heavy normalization. Δ=gold−strict ; positive values indicate degradation.
Corpus
Class
Metric
Gold
No Fill.
Switchboard
b
P
0.558
0.436
b
F1
0.433
0.365
MRDA
fh
R
0.495
0.330
h
R
0.313
0.104
SLUE-HVB
BC
P
0.353
0.092
BC
R
0.750
0.750
TABLE V: Selected class-level effects of lexicon-based removal for deterministic TF-IDF+LR on held-out test splits. P, R, and F1 denote precision, recall, and F1-score. BC denotes SLUE-HVB backchannels.
Spoken Language Understanding (SLU) maps speech to intents, entities, dialogue states, and meaning while accounting for acoustic and conversational context.
Modern automatic speech recognition (ASR) systems trained on extremely large datasets can produce transcripts with numbers written in Arabic numerals. This creates a need for fair comparison with models that output verbatim texts and proper processing of reference transcripts. Popular approaches often reduce text normalization to lowercase and remove punctuation, with no additional normalization applied to languages other than English. In this work, we analyze the impact of normalization of numerical expressions in the evaluation of ASR systems in various languages, using Polish as an example of a highly inflective language. We perform experiments on VoxPopuli and The Polish Parliamentary speech datasets and estimate word error rate (WER) differences for different text normalization approaches. We show that the difference due to the lack of number normalization in WER may be substantial - more than 2 percentage points, and often higher than the differences between systems in popular multilingual benchmarks.
Stanisław Kacprzak, Mieszko Fraś
AGH University of Krakow, Institute of Electronics, Krakow, Poland
Popular ASR test sets adopt inconsistent conventions for numbers, disfluencies, entities, and casing, while standard normalizers erase the format distinctions users care about. Current benchmarks therefore cannot measure whether a model follows user preferences for output style. We introduce PreferenceASR, a test set evaluating ASR systems on their ability to follow natural-language preference instructions across four categories: normalization, entities, disfluencies, and case. Built from seven open-source corpora via a two-stage LLM-assisted pipeline with human verification, it is evaluated with a preference-aware normalizer that selectively skips steps matching the active instruction. Benchmarking four models shows rankings shift across preference types, exposing quality differences traditional evaluation obscures. We publicly release the dataset.
Nithin Rao Koluguri, Sasha Meister, Nikolay Karpov +4
Enterprise conversation analytics asks many questions of millions of interactions. Each question can require reconstructing what people mean and identifying which information matters, repeating costly interpretive work across the same transcripts. We propose a simple principle: clarify the text, then focus the reader. Statement normalization transforms dialogue into short, speaker-attributed statements with source references and semantic tags. The statements make meaning more explicit; the tags support selecting evidence for a particular question. Downstream models can use the full representation or a relevant subset, depending on what helps them make the decision. In an offer-suppression task on customer-service calls, normalization improves a supervised classifier without selection, while weaker prompted readers benefit from both normalization and selection. A small model can learn the normalization contract, while lightweight encoders handle tagging and downstream decisions. Sharing this preparation across questions supports an inference pipeline built entirely from small models, making analytics over millions of conversations substantially less expensive.
Mikhail L. Arbuzov, Karan Dave, Evgeniya Dontsova +5