SMART: Semantic Segment Mapping and Assessment of Recognition Transcripts with Contextual Rubrics
Authors: Xiaoyu Gu, Chencheng Wang, Fuyang Li, Yu Xi, Tao Liu, Jiaoqi Guo, Kai Yu
Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Shanghai Jiao Tong University · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · AISpeech Ltd, Suzhou, China
Word error rate (WER) assigns equal cost to lexical edits despite their different semantic consequences, while overall semantic scores cannot identify which individual errors alter the intended meaning. We formulate ASR semantic evaluation as context-aware, segment-level consequence assessment and propose Semantic Segment Mapping and Assessment of Recognition Transcripts with Contextual Rubrics (SMART), a framework for traceable semantic evaluation and targeted ASR correction. SMART maps reference-hypothesis discrepancies into coherent segments and applies a structured contextual rubric to assign explicit consequence labels. These labels support Semantic Impact Error Rate (SIER), which quantifies consequential errors at utterance, chunk, and lexical levels. Against majority-vote human labels, SMART achieves 0.843 F1 versus 0.745 for direct LLM judgment. Evaluation across multiple LLMs further demonstrates that SMART is relatively insensitive to the choice of evaluation model, while MER and SIER reveal different aspects of ASR quality. A separate target-reference-free rubric guides correction using the target hypothesis and surrounding ASR hypotheses, outperforming a matched w/o Rubric baseline on both datasets, with further absolute reductions of 0.85-0.98 percentage points in MER and 1.18-1.65 percentage points in SIER_lex.
Figures & tables
Figure 1: SMART: Context-aware segment-level semantic consequence assessment and correction. (1) Segment Mapping: Organizes aligned lexical discrepancies into semantically coherent segments. (2) Reference-aware Rubric Evaluation: Judges the semantic consequence of each segment using surrounding reference context and structured criteria. (3) SIER Metric Computation: Quantifies consequential errors at utterance, chunk, and lexical levels. (4) Target-reference-free Post-correction: Uses the target hypothesis and surrounding ASR hypotheses to identify potential semantic errors and guide LLM repair.
ASR System
Utterance-level
Lexical / chunk-level
SER ↓
SIERutt↓
MER ↓
SIERlex↓
SIERchunk↓
Open-Source Models
Qwen3-ASR [ 21 ]
98.07
80.94
12.97
6.08
6.40
whisper-v3 [ 17 ]
94.22
75.59
24.41
16.05
15.16
whisper-v3-turbo [ 17 ]
95.29
76.66
23.19
15.16
14.34
sensevoice [ 1 ]
98.29
82.66
11.87
5.43
6.23
Table 1: ASR evaluation on lexical and semantic. Results on Canvas Speech and TED-LIUM 3 datasets. MER uses Chinese characters and English words as lexical units. All values are percentages; bold indicates the best result within each model group.
Context Why can a longer vacation feel only slightly better?
Reference There are no new memories added; you have not changed the story.
Hypothesis
WER ↓
SIERlex↓
A: There are no new memories added;
8.33
8.33
you have changed the story.
B: Uh, well, there are no new memories
16.67
0.00
added; you have not changed the story.
Table 2: An example of differing lexical and semantic errors. Hypotheses A and B are produced by Qwen3-ASR-Filetrans and Doubao Filetrans, respectively, for the same utterance from TED-LIUM release 3. A omits the consequential negation “not,” whereas B preserves it but introduces inconsequential fillers. WER favors A, while SIERlex favors B. All values are percentages; lower is better.
Classifier
Acc. ↑
Prec. ↑
Rec. ↑
F1 ↑
Similarity-based metrics
BERTScore [ 26 ]
0.215
0.214
0.910
0.331
SeMaScore [ 20 ]
0.370
0.261
0.910
0.384
LLM-based evaluation
Interactive ASR [ 23 ]
0.515
0.297
0.955
0.440
Direct LLM
0.900
0.764
0.731
0.745
Table 3: Agreement with human semantic consequence judgments. Classification results on discrepant segments from Canvas Speech and TED-LIUM 3, using majority-vote human annotations as ground truth. Consequential is the positive class. Acc., Prec., Rec., and F1 denote accuracy, precision, recall, and F1-score, respectively. All LLM-based methods use DeepSeek-V4-Flash. Bold indicates the highest value in each column.
Figure 2: Target-reference-free correction of SenseVoice on Canvas Speech and TED-LIUM 3. Bars report MER and SIERlex for the original hypothesis, the matched w/o Rubric baseline, and rubric-guided correction. Pink caps indicate newly introduced error mass, while double-headed arrows indicate repaired error mass relative to the original hypothesis.
Evaluation LLM
Acc. ↑
Prec. ↑
Rec. ↑
F1 ↑
GPT-4.1
0.895
0.750
0.800
0.774
GPT-5.5
0.915
0.868
0.733
0.795
GPT-5.6-Luna
0.895
0.714
0.889
0.792
DeepSeek-V4-Flash
0.925
0.857
0.830
0.843
Table 4: Sensitivity of SMART to the evaluation LLM. Semantic consequence classification on the same human-annotated segments using different evaluation LLMs. Consequential is the positive class. Acc., Prec., Rec., and F1 denote accuracy, precision, recall, and F1-score, respectively. Bold indicates the highest value in each column.
Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metrics tested, that the best-performing SemDist configurations achieve the highest overall agreement, ahead of CER and BERTScore, and that no single model is best across settings. CER, despite its simplicity and low cost, remains remarkably close to these best configurations. In line with prior recommendations, our results support shifting ASR evaluation toward CER both for English and for morphosyllabic writing systems as it is a more interpretable and low-cost metric for what evaluation should actually capture, and using SemDist as a complementary evaluation.
Automatic Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning. Embedding-based semantic metrics are better correlated with human perception, but decoder-based Large Language Models (LLMs) remain underexplored for this task. This paper evaluates their relevance through three approaches: (1) selecting the best hypothesis between two candidates, (2) computing semantic distance using generative embeddings, and (3) qualitative classification of errors. On the HATS dataset, the best LLMs achieve 92--94% agreement with human annotators for hypothesis selection, compared to 63% for WER, also outperforming semantic metrics. Embeddings from decoder-based LLMs show performance comparable to encoder models. Finally, LLMs offer a promising direction for interpretable and semantic ASR evaluation.
Recent years have witnessed remarkable progress in automatic speech recognition (ASR), driven by advances in model architectures and large-scale training data. However, two important aspects remain underexplored. First, Word Error Rate (WER), the dominant evaluation metric for decades, treats all words equally and often fails to reflect the semantic correctness of an utterance at the sentence level. Second, interactive correction-an essential component of human communication-has rarely been systematically studied in ASR research. In this paper, we integrate these two perspectives under an agentic framework for interactive ASR. We propose leveraging LLM-as-a-Judge as a semantic-aware evaluation metric to assess recognition quality beyond token-level accuracy. Furthermore, we design an LLM-driven agent framework to simulate human-like multi-turn interaction, enabling iterative refinement of recognition outputs through semantic feedback. Extensive experiments are conducted on standard benchmarks, including GigaSpeech (English), WenetSpeech (Chinese), the ASRU 2019 code-switching test set. Both objective and subjective evaluations demonstrate the effectiveness of the proposed framework in improving semantic fidelity and interactive correction capability. We will release the code to facilitate future research in interactive and agentic ASR.
Peng Wang, Yanqiao Zhu, Zixuan Jiang +8
The Chinese University of Hong Kong, Shenzhen · X-LANCE Lab, Shanghai Jiao Tong University · Xi’an Jiaotong University +2