SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation

Mingzhe Lu; Jinhao Liu; Yunpeng Li; Hang Su; Lichun Fan; Jian Luan

cs.CLOct 15, 2026

SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation

Authors: Mingzhe Lu, Jinhao Liu, Yunpeng Li, Hang Su, Lichun Fan, Jian Luan

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Xiaomi Inc.

Abstract

In automatic speech recognition (ASR), transcription errors with similar lexical or semantic discrepancies can have markedly different consequences for the situation conveyed by speech. Capturing these differences requires not merely contextual assessment, but an understanding of what the spoken discourse describes, establishes, or seeks to bring about. However, relying on LLM judges introduces computational cost and scoring variability. We introduce SITAR, a situational distortion metric for ASR, realized through a comprehensive end-to-end construction framework. Specifically, this framework aggregates situational severity and cumulative extent into a unified score via a bounded recursive formulation. Using targets derived from this formulation with limited offline LLM support, we train SITAR as a lightweight BERT-family regressor that scores reference-hypothesis pairs in a single forward pass, entirely free of any prohibitive runtime costs. Recognizing that introducing a new metric demands rigorous justification, we formalize a principled validation protocol spanning both extrinsic validity, assessed at macro and micro levels, and intrinsic scorer reliability. The former, evaluated across diverse datasets and models, demonstrates that SITAR reliably tracks broad capability scaling while providing orthogonal diagnostic utility, whereas the latter, examined on a purpose-built controlled benchmark, confirms the operational soundness of the scoring instrument itself. Through systematic validation, we demonstrate that SITAR holds strong potential as a practical, lightweight complement to existing ASR metrics. We will release the full recipes (code, model weights, dataset, etc.) after the internal review to support community research.

Figures & tables

Figure 1: Overview of Sitar construction: CGD extracts paired gaps from transcripts, which are annotated for situational severity and aggregated into supervision targets to train the LLM-free scorer.

Model Name	Dataset	WER	NE-WER	BERTScore F1	SemDist	AER	Sitar
whisper-tiny	fleurs_en	0.1446	0.6500	0.9437	0.0711	0.2478	0.3872
whisper-base		0.1150	0.5500	0.9516	0.0558	0.2015	0.2999
whisper-small		0.0711	0.4250	0.9585	0.0457	0.1586	0.2113
whisper-medium		0.0671	0.3167	0.9610	0.0421	0.1508	0.1590
whisper-large		0.0465	0.3000	0.9621	0.0410	0.1477	0.1388
wav2vec2-base		0.2016	0.7750	0.8709	0.0855	0.2393	0.3670

Table 1: Comparison of ASR evaluation metrics across datasets.

Construction profile	Mean	Median	Standard deviation	Central 50%	IQR
$\mathcal{T}_{0}$	0.0813	0.0763	0.0491	[0.0575, 0.0986]	0.0411
$2\mathcal{T}_{0}$	0.1268	0.0946	0.0791	[0.0848, 0.1269]	0.0420
$\mathcal{T}_{1}$	0.5175	0.5131	0.1634	[0.3907, 0.6414]	0.2506
$\mathcal{T}_{1}+\mathcal{T}_{0}$	0.5181	0.5032	0.1629	[0.3873, 0.6512]	0.2639
$\mathcal{T}_{1}+2\mathcal{T}_{0}$	0.5324	0.5268	0.1672	[0.3921, 0.6740]	0.2819
$\mathcal{T}_{2}$	0.6612	0.6807	0.1344	[0.5711, 0.7602]	0.1891

Table 2: Cross-reference distributions of normalized Sitar scores.

Table 4

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Highest nominal tier	Base $\beta$	Score interval
$\mathcal{T}_{0}$	$0$	$[0,1/3)$
	$1/3$	$[1/3,2/3)$
$\mathcal{T}_{1}$	$2/3$	$[2/3,1)$
	$1$	$[1,4/3)$
	$4/3$	$[4/3,5/3)$
$\mathcal{T}_{2}$	$5/3$	$[5/3,2)$

Appendix

Table 5: Base positions and score intervals for finite gap counts. The highest nominal tier is determined by the reviewed assignments.

Gaps $n$	Refinement $R_{n}$	Score increment $R_{n}/3$
1	0.000000	0.000000
2	0.318708	0.106236
3	0.535841	0.178614
4	0.683772	0.227924
5	0.784557	0.261519
6	0.853220	0.284407

Appendix

Table 6: Quantity refinement for

n

gaps at the same nominal tier, with

\rho=10^{-1/6}

. Values are rounded to six decimal places.

Gap	Original labels	Majority tier	Reviewed tier	Position $b_{i}$
$g_{0}$	$(0,0,0)$	0	0	$0$
$g_{1}$	$(0,0,1)$	0	0	$0$
$g_{2}$	$(1,1,0)$	1	1	$1$
$g_{3}$	$(2,2,1)$	2	2	$2$
$g_{4}$	$(1,1,2)$	1	2	$5/3$

Appendix

Table 7: Illustrative gap annotations for the mixed-tier calculation. The three original labels determine the majority tier, and the reviewed tier determines the nominal count.

Type	Source	N	Mean	P25	P50	P75	P95
Audio	AppTek_Clip	31,354	10.77 s	3.14 s	7.66 s	15.90 s	31.79 s
Audio	FLEURS_en	3,643	10.19 s	7.68 s	9.66 s	11.88 s	16.90 s
Transcript	AppTek_Clip	31,354	24.26	8	18	34	70
Transcript	FLEURS_en	3,643	21.21	16	20	25	34

Appendix

Table 8: Statistics of candidate audio and reference transcripts across AppTek and FLEURS datasets.

Source	Locale	Qwen3	Granite	Whisper	Parakeet	Total
AppTek_Clip	en-AU	188	190	167	155	700
	en-CA	151	157	150	187	645
	en-CN	185	151	160	177	673
	en-GB	162	167	177	152	658
	en-IE	184	165	171	151	671
	en-MX	176	164	167	155	662

Appendix

Table 9: Selected reference–hypothesis pairs by source, locale, and ASR system.

Figure 2: Domain distribution of the 6,038 selected AppTek pairs.

Figure 3: Distribution of unique training pairs across supervision score bins (width 0.1). Hatched and solid blue regions denote original and added pairs, respectively. Values above bars show total counts; labels inside (‘+’ prefixed) indicate added counts.

Figure 4: Normalized supervision scores across all 12,347 pairs. The upper panel shows the sample count at each score, and the lower panel marks the positions of the 246 distinct values. Distinct values are counted by rounding raw scores to four decimal places before normalization. Dashed lines mark the normalized base-score interval boundaries.

Component	Setting
Encoder	DeBERTa-v3-large; full fine-tuning
Input and maximum length	Original reference–hypothesis pair; 512 tokens
Pooling	One-layer bidirectional LSTM, followed by masked mean
Prediction heads	Linear scalar head and six-logit ordinal head
Head dropout	0.1
Loss weights $(\lambda_{\mathrm{mse}},\lambda_{\mathrm{boundary}},\lambda_{\mathrm{ordinal}})$	$(1.0,0.2,0.5)$

Appendix

Table 10: Configuration of the final Sitar scorer. Learning rates distinguish the pretrained encoder from the LSTM and prediction heads.

Pooling	MSE $\downarrow$	MAE $\downarrow$	Pearson $r$ $\uparrow$	Spearman $\rho_{s}$ $\uparrow$
CLS	0.138204	0.261133	0.810322	0.813424
Mean	0.143248	0.277111	0.795377	0.793949
Last-four-layer concatenation	0.150199	0.275274	0.794139	0.789104
Learned layer mixture	0.147843	0.267151	0.798982	0.796896
LSTM	0.129790	0.255395	0.812736	0.807285

Appendix

Table 11: Pooling comparison with MSE-only training and seed 42. Bold values indicate the best result in each column.

$\lambda_{\mathrm{boundary}}$	$\lambda_{\mathrm{ordinal}}$	MSE $\downarrow$	Spearman $\rho_{s}$ $\uparrow$
0.0	0.0	$0.134565\pm 0.004427$	$0.802046\pm 0.002034$
0.0	0.1	$0.131830\pm 0.001990$	$0.802677\pm 0.002407$
0.0	0.9	$0.131293\pm 0.001865$	$0.808193\pm 0.003262$
0.3	0.0	$0.133434\pm 0.002802$	$0.803971\pm 0.005725$
0.3	0.9	$0.130475\pm 0.002781$	$0.806029\pm 0.006550$
0.2	0.5	$\mathbf{0.127040}\pm 0.004466$	$\mathbf{0.808929}\pm 0.005065$

Appendix

Table 12: Loss-weight comparison with LSTM pooling and full encoder fine-tuning. Results are mean

\pm

sample standard deviation over five seeds;

\lambda_{\mathrm{mse}}=1

throughout. Bold marks the best means.

Input representation; encoder fully trainable
Configuration	MSE $\downarrow$	Spearman $\rho_{s}$ $\uparrow$
Original reference–hypothesis pair	$\mathbf{0.127040}\pm 0.004466$	$\mathbf{0.808929}\pm 0.005065$
Inline gap markers	$0.136698\pm 0.002313$	$0.802945\pm 0.003336$
Appended gap spans	$0.140356\pm 0.005461$	$0.798801\pm 0.003963$
Encoder adaptation; original transcript input
Frozen encoder	$0.161915\pm 0.002660$	$0.764525\pm 0.004795$

Appendix

Table 13: Input and encoder-adaptation ablations. Each row reports five-seed mean

\pm

sample standard deviation with LSTM pooling and loss weights

(1,0.2,0.5)

. The original-input and full-fine-tuning rows reuse the same baseline runs.

Encoder learning rate	$5\times 10^{-5}$	$10^{-4}$	$2\times 10^{-4}$	$4\times 10^{-4}$
$3\times 10^{-6}$	0.134568	0.133172	0.133939	0.134907
$6\times 10^{-6}$	0.128956	0.123872	0.124715	0.129092
$10^{-5}$	0.127830	0.130582	0.128878	0.128369
$2\times 10^{-5}$	0.136164	0.138593	0.135775	0.132319

Appendix

Table 14: Learning-rate sensitivity: mean MSE over seeds 42 and 43. Columns specify the learning rate for pooling and prediction heads. The bold entry is the lowest mean.

Ordinal thresholds	$q$	MSE $\downarrow$	Spearman $\rho_{s}$ $\uparrow$
Six thresholds	2	$\mathbf{0.123872}\pm 0.002990$	$0.811107\pm 0.006701$
Six thresholds	3	$0.131291\pm 0.000161$	$0.805553\pm 0.001183$
Six thresholds	4	$0.129176\pm 0.003418$	$\mathbf{0.811186}\pm 0.002762$
Two thresholds	2	$0.134309\pm 0.002946$	$0.800777\pm 0.009875$
Two thresholds	3	$0.131726\pm 0.000921$	$0.802823\pm 0.008857$
Two thresholds	4	$0.133642\pm 0.004875$	$0.804636\pm 0.008837$

Appendix

Table 15: Ordinal-threshold and boundary-resolution comparison. Results are mean

\pm

sample standard deviation over seeds 42 and 43. The six-threshold,

q=2

row reuses the learning-rate baseline.

Positive-class weights	Uniform	Low	High	Middle
$\bm{p}_{0}$	0.127040	0.131188	0.127591	0.132335
$\bm{p}_{1}$	0.128455	0.128698	0.127074	0.127770
$\bm{p}_{2}$	0.128300	0.131183	0.126756	0.127376
$\bm{p}_{3}$	0.128768	0.129774	0.128308	0.127996

Appendix

Table 16: Ordinal reweighting: five-seed mean MSE. Columns indicate threshold-loss weighting; rows indicate positive-class weighting. The bold entry is the lowest mean.

Seed	MSE $\downarrow$	MAE $\downarrow$	Pearson $r$ $\uparrow$	Spearman $\rho_{s}$ $\uparrow$
42	0.126302	0.254662	0.818037	0.807938
43	0.129235	0.255753	0.811441	0.804729
44	0.130549	0.250619	0.810363	0.804272
45	0.127689	0.244722	0.819069	0.808274
46	0.124689	0.245155	0.821666	0.815188

Appendix

Table 17: Independent retraining of the final scorer configuration. All results use the same 1,292-pair test split and raw predictions. Bold values identify the best result in each column.

Dataset	AER span	Sitar span	Span ratio	$\rho_{s}$
FLEURS	0.1001	0.2484	2.48	1.0000
LibriSpeech	0.0469	0.1623	3.46	0.9643
VoxPopuli	0.0394	0.2400	6.09	$-0.1071$

Appendix

Table 18: Score separation and rank agreement derived from the seven systems per dataset in Table 1 . A span is the maximum score minus the minimum score. The span ratio is Sitar divided by AER, and

\rho_{s}

is their Spearman rank correlation.

Domain	References	Candidate gaps	Transcript pairs
Culture	9	36	99
Finance	4	16	44
Services	7	28	77
Public affairs	5	20	55
General communication	5	20	55
Total	30	120	330

Appendix

Table 19: Composition of the controlled dataset. Each reference has four candidate gaps for annotation and 11 hypothesis variants for evaluation. Culture includes cultural and sports content.

Gap	Reference span	Hypothesis span	Tier
$G_{0}$	$\emptyset$	I	$\mathcal{T}_{0}$
$G_{1}$	The figures	Most of the figures	$\mathcal{T}_{1}$
$G_{2}$	Do not send it to the client until I approve it.	You can send it to the client before I approve it.	$\mathcal{T}_{2}$
$G_{3}$	$\emptyset$	uh,	$\mathcal{T}_{0}$

Appendix

Table 20: The four fixed edits in reference 27. The symbol

\emptyset

denotes an insertion.

G_{0}

is inserted at the beginning of the reference, and

G_{3}

immediately before “meantime.” Gap IDs indicate positions, independently of severity.

Variant	Construction profile	Retained edits	Sitar
$a$	$\mathcal{T}_{0}$	$G_{0}$	0.0274
$b$	$2\mathcal{T}_{0}$	$G_{0},G_{3}$	0.0564
$c$	$\mathcal{T}_{1}$	$G_{1}$	0.3204
$d$	$\mathcal{T}_{1}+\mathcal{T}_{0}$	$G_{0},G_{1}$	0.3356
$e$	$\mathcal{T}_{1}+2\mathcal{T}_{0}$	$G_{0},G_{1},G_{3}$	0.3774
$f$	$\mathcal{T}_{2}$	$G_{2}$	0.7573

Appendix

Table 21: All 11 variants of reference 27. Each hypothesis applies exactly the listed edits to the reference. The final column reports normalized Sitar predictions, rounded to four decimal places. These construction profiles are used for all 30 references.

	B $\mathcal{T}_{0}$	B $\mathcal{T}_{1}$	B $\mathcal{T}_{2}$	Total
A $\mathcal{T}_{0}$	58	0	0	58
A $\mathcal{T}_{1}$	3	24	3	30
A $\mathcal{T}_{2}$	2	4	26	32
Total	63	28	29	120

Appendix

Table 22: Human annotation confusion matrix before adjudication. Rows give annotator A’s labels and columns give annotator B’s labels for the same 120 gaps.

Distinguishing tier	Comparisons	Correct	Agreement
$\mathcal{T}_{2}$	870	793	91.15%
$\mathcal{T}_{1}$	471	393	83.44%
$\mathcal{T}_{0}$	306	199	65.03%
Total	1,647	1,385	84.09%

Appendix

Table 23: Ordering agreement by the highest tier with a count difference. Counts use final adjudicated labels. Agreement is the fraction of correct strict comparisons within each row.

ASR Evaluation

Automatic speech recognition (ASR) evaluation measures transcript accuracy, robustness, fairness, and usefulness across languages, speakers, and real-world conditions.

See all 145 papers

Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER
Sep 18, 2026Hritika Sharma, Thibault Bañeras-Roux, Alessandra Pinto +4ASR Evaluation Semantic Textual Similarity
Faculty of Computer Science and Technology, Algoma University · Idiap Research Institute · School of Psychology, Algoma University +1
Beyond Word Error Rate: Auditing the Diversity Tax in Speech Recognition through Dataset Cartography
Mar 5, 2026Ting-Hui Cheng, Line H. Clemmensen, Sneha DasASR Evaluation Demographic Bias in ASR
Department of Applied Mathematics and Computer Science, Technical University of Denmark, Denmark · Department of Mathematical Sciences, University of Copenhagen, Denmark
Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy
Jun 3, 2026Zhihan Li, Hankun Wang, Yiwei Guo +3ASR Evaluation Automatic Speech Recognition
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China · MoE Key Lab of Artificial Intelligence, Jiangsu Key Lab of Language Computing, China

Sep 18, 2026J/K move · Enter open · S save

Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER

Hritika Sharma, Thibault Bañeras-Roux, Alessandra Pinto, Petr Motlicek+3

Faculty of Computer Science and Technology, Algoma University · Idiap Research Institute · School of Psychology, Algoma University+1

ASR Evaluation Semantic Textual Similarity

Open paper AskPDF arXiv

Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metrics tested, that the best-performing SemDist configurations achieve the highest overall agreement, ahead of CER and BERTScore, and that no single model is best across settings. CER, despite its simplicity and low cost, remains remarkably close to these best configurations. In line with prior recommendations, our results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.

Abstract

ASR Evaluation

Explore similar work

Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER

Beyond Word Error Rate: Auditing the Diversity Tax in Speech Recognition through Dataset Cartography

Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy