SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation
Authors: Mingzhe Lu, Jinhao Liu, Yunpeng Li, Hang Su, Lichun Fan, Jian Luan
Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Xiaomi Inc.
In automatic speech recognition (ASR), transcription errors with similar lexical or semantic discrepancies can have markedly different consequences for the situation conveyed by speech. Capturing these differences requires not merely contextual assessment, but an understanding of what the spoken discourse describes, establishes, or seeks to bring about. However, relying on LLM judges introduces computational cost and scoring variability. We introduce SITAR, a situational distortion metric for ASR, realized through a comprehensive end-to-end construction framework. Specifically, this framework aggregates situational severity and cumulative extent into a unified score via a bounded recursive formulation. Using targets derived from this formulation with limited offline LLM support, we train SITAR as a lightweight BERT-family regressor that scores reference-hypothesis pairs in a single forward pass, entirely free of any prohibitive runtime costs. Recognizing that introducing a new metric demands rigorous justification, we formalize a principled validation protocol spanning both extrinsic validity, assessed at macro and micro levels, and intrinsic scorer reliability. The former, evaluated across diverse datasets and models, demonstrates that SITAR reliably tracks broad capability scaling while providing orthogonal diagnostic utility, whereas the latter, examined on a purpose-built controlled benchmark, confirms the operational soundness of the scoring instrument itself. Through systematic validation, we demonstrate that SITAR holds strong potential as a practical, lightweight complement to existing ASR metrics. We will release the full recipes (code, model weights, dataset, etc.) after the internal review to support community research.
Figures & tables
Figure 1: Overview of Sitar construction: CGD extracts paired gaps from transcripts, which are annotated for situational severity and aggregated into supervision targets to train the LLM-free scorer.
Model Name
Dataset
WER
NE-WER
BERTScore F1
SemDist
AER
Sitar
whisper-tiny
fleurs_en
0.1446
0.6500
0.9437
0.0711
0.2478
0.3872
whisper-base
0.1150
0.5500
0.9516
0.0558
0.2015
0.2999
whisper-small
0.0711
0.4250
0.9585
0.0457
0.1586
0.2113
whisper-medium
0.0671
0.3167
0.9610
0.0421
0.1508
0.1590
whisper-large
0.0465
0.3000
0.9621
0.0410
0.1477
0.1388
wav2vec2-base
0.2016
0.7750
0.8709
0.0855
0.2393
0.3670
Table 1: Comparison of ASR evaluation metrics across datasets.
Construction profile
Mean
Median
Standard deviation
Central 50%
IQR
T0
0.0813
0.0763
0.0491
[0.0575, 0.0986]
0.0411
2T0
0.1268
0.0946
0.0791
[0.0848, 0.1269]
0.0420
T1
0.5175
0.5131
0.1634
[0.3907, 0.6414]
0.2506
T1+T0
0.5181
0.5032
0.1629
[0.3873, 0.6512]
0.2639
T1+2T0
0.5324
0.5268
0.1672
[0.3921, 0.6740]
0.2819
T2
0.6612
0.6807
0.1344
[0.5711, 0.7602]
0.1891
Table 2: Cross-reference distributions of normalized Sitar scores.
Table 4
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Highest nominal tier
Base β
Score interval
T0
0
[0,1/3)
1/3
[1/3,2/3)
T1
2/3
[2/3,1)
1
[1,4/3)
4/3
[4/3,5/3)
T2
5/3
[5/3,2)
Appendix
Table 5: Base positions and score intervals for finite gap counts. The highest nominal tier is determined by the reviewed assignments.
Gaps n
Refinement Rn
Score increment Rn/3
1
0.000000
0.000000
2
0.318708
0.106236
3
0.535841
0.178614
4
0.683772
0.227924
5
0.784557
0.261519
6
0.853220
0.284407
Appendix
Table 6: Quantity refinement for n gaps at the same nominal tier, with ρ=10−1/6 . Values are rounded to six decimal places.
Gap
Original labels
Majority tier
Reviewed tier
Position bi
g0
(0,0,0)
0
0
0
g1
(0,0,1)
0
0
0
g2
(1,1,0)
1
1
1
g3
(2,2,1)
2
2
2
g4
(1,1,2)
1
2
5/3
Appendix
Table 7: Illustrative gap annotations for the mixed-tier calculation. The three original labels determine the majority tier, and the reviewed tier determines the nominal count.
Type
Source
N
Mean
P25
P50
P75
P95
Audio
AppTek_Clip
31,354
10.77 s
3.14 s
7.66 s
15.90 s
31.79 s
FLEURS_en
3,643
10.19 s
7.68 s
9.66 s
11.88 s
16.90 s
Transcript
AppTek_Clip
31,354
24.26
8
18
34
70
FLEURS_en
3,643
21.21
16
20
25
34
Appendix
Table 8: Statistics of candidate audio and reference transcripts across AppTek and FLEURS datasets.
Source
Locale
Qwen3
Granite
Whisper
Parakeet
Total
AppTek_Clip
en-AU
188
190
167
155
700
en-CA
151
157
150
187
645
en-CN
185
151
160
177
673
en-GB
162
167
177
152
658
en-IE
184
165
171
151
671
en-MX
176
164
167
155
662
Appendix
Table 9: Selected reference–hypothesis pairs by source, locale, and ASR system.
Figure 2: Domain distribution of the 6,038 selected AppTek pairs.
Figure 3: Distribution of unique training pairs across supervision score bins (width 0.1). Hatched and solid blue regions denote original and added pairs, respectively. Values above bars show total counts; labels inside (‘+’ prefixed) indicate added counts.
Figure 4: Normalized supervision scores across all 12,347 pairs. The upper panel shows the sample count at each score, and the lower panel marks the positions of the 246 distinct values. Distinct values are counted by rounding raw scores to four decimal places before normalization. Dashed lines mark the normalized base-score interval boundaries.
Component
Setting
Encoder
DeBERTa-v3-large; full fine-tuning
Input and maximum length
Original reference–hypothesis pair; 512 tokens
Pooling
One-layer bidirectional LSTM, followed by masked mean
Prediction heads
Linear scalar head and six-logit ordinal head
Head dropout
0.1
Loss weights (λmse,λboundary,λordinal)
(1.0,0.2,0.5)
Appendix
Table 10: Configuration of the final Sitar scorer. Learning rates distinguish the pretrained encoder from the LSTM and prediction heads.
Pooling
MSE ↓
MAE ↓
Pearson r↑
Spearman ρs↑
CLS
0.138204
0.261133
0.810322
0.813424
Mean
0.143248
0.277111
0.795377
0.793949
Last-four-layer concatenation
0.150199
0.275274
0.794139
0.789104
Learned layer mixture
0.147843
0.267151
0.798982
0.796896
LSTM
0.129790
0.255395
0.812736
0.807285
Appendix
Table 11: Pooling comparison with MSE-only training and seed 42. Bold values indicate the best result in each column.
λboundary
λordinal
MSE ↓
Spearman ρs↑
0.0
0.0
0.134565±0.004427
0.802046±0.002034
0.0
0.1
0.131830±0.001990
0.802677±0.002407
0.0
0.9
0.131293±0.001865
0.808193±0.003262
0.3
0.0
0.133434±0.002802
0.803971±0.005725
0.3
0.9
0.130475±0.002781
0.806029±0.006550
0.2
0.5
0.127040±0.004466
0.808929±0.005065
Appendix
Table 12: Loss-weight comparison with LSTM pooling and full encoder fine-tuning. Results are mean ± sample standard deviation over five seeds; λmse=1 throughout. Bold marks the best means.
Configuration
MSE ↓
Spearman ρs↑
Input representation; encoder fully trainable
Original reference–hypothesis pair
0.127040±0.004466
0.808929±0.005065
Inline gap markers
0.136698±0.002313
0.802945±0.003336
Appended gap spans
0.140356±0.005461
0.798801±0.003963
Encoder adaptation; original transcript input
Frozen encoder
0.161915±0.002660
0.764525±0.004795
Appendix
Table 13: Input and encoder-adaptation ablations. Each row reports five-seed mean ± sample standard deviation with LSTM pooling and loss weights (1,0.2,0.5) . The original-input and full-fine-tuning rows reuse the same baseline runs.
Encoder learning rate
5×10−5
10−4
2×10−4
4×10−4
3×10−6
0.134568
0.133172
0.133939
0.134907
6×10−6
0.128956
0.123872
0.124715
0.129092
10−5
0.127830
0.130582
0.128878
0.128369
2×10−5
0.136164
0.138593
0.135775
0.132319
Appendix
Table 14: Learning-rate sensitivity: mean MSE over seeds 42 and 43. Columns specify the learning rate for pooling and prediction heads. The bold entry is the lowest mean.
Ordinal thresholds
q
MSE ↓
Spearman ρs↑
Six thresholds
2
0.123872±0.002990
0.811107±0.006701
Six thresholds
3
0.131291±0.000161
0.805553±0.001183
Six thresholds
4
0.129176±0.003418
0.811186±0.002762
Two thresholds
2
0.134309±0.002946
0.800777±0.009875
Two thresholds
3
0.131726±0.000921
0.802823±0.008857
Two thresholds
4
0.133642±0.004875
0.804636±0.008837
Appendix
Table 15: Ordinal-threshold and boundary-resolution comparison. Results are mean ± sample standard deviation over seeds 42 and 43. The six-threshold, q=2 row reuses the learning-rate baseline.
Positive-class weights
Uniform
Low
High
Middle
p0
0.127040
0.131188
0.127591
0.132335
p1
0.128455
0.128698
0.127074
0.127770
p2
0.128300
0.131183
0.126756
0.127376
p3
0.128768
0.129774
0.128308
0.127996
Appendix
Table 16: Ordinal reweighting: five-seed mean MSE. Columns indicate threshold-loss weighting; rows indicate positive-class weighting. The bold entry is the lowest mean.
Seed
MSE ↓
MAE ↓
Pearson r↑
Spearman ρs↑
42
0.126302
0.254662
0.818037
0.807938
43
0.129235
0.255753
0.811441
0.804729
44
0.130549
0.250619
0.810363
0.804272
45
0.127689
0.244722
0.819069
0.808274
46
0.124689
0.245155
0.821666
0.815188
Appendix
Table 17: Independent retraining of the final scorer configuration. All results use the same 1,292-pair test split and raw predictions. Bold values identify the best result in each column.
Dataset
AER span
Sitar span
Span ratio
ρs
FLEURS
0.1001
0.2484
2.48
1.0000
LibriSpeech
0.0469
0.1623
3.46
0.9643
VoxPopuli
0.0394
0.2400
6.09
−0.1071
Appendix
Table 18: Score separation and rank agreement derived from the seven systems per dataset in Table 1 . A span is the maximum score minus the minimum score. The span ratio is Sitar divided by AER, and ρs is their Spearman rank correlation.
Domain
References
Candidate gaps
Transcript pairs
Culture
9
36
99
Finance
4
16
44
Services
7
28
77
Public affairs
5
20
55
General communication
5
20
55
Total
30
120
330
Appendix
Table 19: Composition of the controlled dataset. Each reference has four candidate gaps for annotation and 11 hypothesis variants for evaluation. Culture includes cultural and sports content.
Gap
Reference span
Hypothesis span
Tier
G0
∅
I
T0
G1
The figures
Most of the figures
T1
G2
Do not send it to the client until I approve it.
You can send it to the client before I approve it.
T2
G3
∅
uh,
T0
Appendix
Table 20: The four fixed edits in reference 27. The symbol ∅ denotes an insertion. G0 is inserted at the beginning of the reference, and G3 immediately before “meantime.” Gap IDs indicate positions, independently of severity.
Variant
Construction profile
Retained edits
Sitar
a
T0
G0
0.0274
b
2T0
G0,G3
0.0564
c
T1
G1
0.3204
d
T1+T0
G0,G1
0.3356
e
T1+2T0
G0,G1,G3
0.3774
f
T2
G2
0.7573
Appendix
Table 21: All 11 variants of reference 27. Each hypothesis applies exactly the listed edits to the reference. The final column reports normalized Sitar predictions, rounded to four decimal places. These construction profiles are used for all 30 references.
B T0
B T1
B T2
Total
A T0
58
0
0
58
A T1
3
24
3
30
A T2
2
4
26
32
Total
63
28
29
120
Appendix
Table 22: Human annotation confusion matrix before adjudication. Rows give annotator A’s labels and columns give annotator B’s labels for the same 120 gaps.
Distinguishing tier
Comparisons
Correct
Agreement
T2
870
793
91.15%
T1
471
393
83.44%
T0
306
199
65.03%
Total
1,647
1,385
84.09%
Appendix
Table 23: Ordering agreement by the highest tier with a count difference. Counts use final adjudicated labels. Agreement is the fraction of correct strict comparisons within each row.
Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metrics tested, that the best-performing SemDist configurations achieve the highest overall agreement, ahead of CER and BERTScore, and that no single model is best across settings. CER, despite its simplicity and low cost, remains remarkably close to these best configurations. In line with prior recommendations, our results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.
Automatic speech recognition (ASR) systems are predominantly evaluated using the Word Error Rate (WER). However, raw token-level metrics fail to capture semantic fidelity and routinely obscures the `diversity tax', the disproportionate burden on marginalized and atypical speaker due to systematic recognition failures. In this paper, we explore the limitations of relying solely on lexical counts by systematically evaluating a broader class of non-linear and semantic metrics. To enable rigorous model auditing, we introduce the sample difficulty index (SDI), a novel metric that quantifies how intrinsic demographic and acoustic factors drive model failure. By mapping SDI on data cartography, we demonstrate that metrics EmbER and SemDist expose hidden systemic biases and inter-model disagreements that WER ignores. Finally, our findings are the first steps towards a robust audit framework for prospective safety analysis, empowering developers to audit and mitigate ASR disparities prior to deployment.
Ting-Hui Cheng, Line H. Clemmensen, Sneha Das
Department of Applied Mathematics and Computer Science, Technical University of Denmark, Denmark · Department of Mathematical Sciences, University of Copenhagen, Denmark
Automatic speech recognition systems commonly rely on reference transcriptions for evaluation, while reference-free approaches often depend on internal confidence estimation or auxiliary language models. We propose READ (Reference-free Hypothesis Evaluation with Acoustic Discrepancy), a novel metric that evaluates ASR hypotheses directly from the speech signal. READ emphasizes the acoustic grounding of hypotheses. It uses a pretrained auto-regressive TTS model to compute the conditional likelihood of speech tokens given a text hypothesis, to measure fine-grained acoustic discrepancy between speech and text. Without additional training, READ can be applied for hypothesis refinement. Experiments show that READ correlates with specific recognition errors and improves ASR outputs, achieving up to 20% relative error rate reduction, with particularly strong gains under noisy conditions.
Zhihan Li, Hankun Wang, Yiwei Guo +3
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China