SMART: Semantic Segment Mapping and Assessment of Recognition Transcripts with Contextual Rubrics
Authors: Xiaoyu Gu, Chencheng Wang, Fuyang Li, Yu Xi, Tao Liu, Jiaoqi Guo, Kai Yu
Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Shanghai Jiao Tong University · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · AISpeech Ltd, Suzhou, China
Word error rate (WER) assigns equal cost to lexical edits despite their different semantic consequences, while overall semantic scores cannot identify which individual errors alter the intended meaning. We formulate ASR semantic evaluation as context-aware, segment-level consequence assessment and propose Semantic Segment Mapping and Assessment of Recognition Transcripts with Contextual Rubrics (SMART), a framework for traceable semantic evaluation and targeted ASR correction. SMART maps reference-hypothesis discrepancies into coherent segments and applies a structured contextual rubric to assign explicit consequence labels. These labels support Semantic Impact Error Rate (SIER), which quantifies consequential errors at utterance, chunk, and lexical levels. Against majority-vote human labels, SMART achieves 0.843 F1 versus 0.745 for direct LLM judgment. Evaluation across multiple LLMs further demonstrates that SMART is relatively insensitive to the choice of evaluation model, while MER and SIER reveal different aspects of ASR quality. A separate target-reference-free rubric guides correction using the target hypothesis and surrounding ASR hypotheses, outperforming a matched w/o Rubric baseline on both datasets, with further absolute reductions of 0.85-0.98 percentage points in MER and 1.18-1.65 percentage points in SIER_lex.
Figures & tables
Figure 1: SMART: Context-aware segment-level semantic consequence assessment and correction. (1) Segment Mapping: Organizes aligned lexical discrepancies into semantically coherent segments. (2) Reference-aware Rubric Evaluation: Judges the semantic consequence of each segment using surrounding reference context and structured criteria. (3) SIER Metric Computation: Quantifies consequential errors at utterance, chunk, and lexical levels. (4) Target-reference-free Post-correction: Uses the target hypothesis and surrounding ASR hypotheses to identify potential semantic errors and guide LLM repair.
ASR System
Utterance-level
Lexical / chunk-level
SER ↓
SIERutt↓
MER ↓
SIERlex↓
SIERchunk↓
Open-Source Models
Qwen3-ASR [ 21 ]
98.07
80.94
12.97
6.08
6.40
whisper-v3 [ 17 ]
94.22
75.59
24.41
16.05
15.16
whisper-v3-turbo [ 17 ]
95.29
76.66
23.19
15.16
14.34
sensevoice [ 1 ]
98.29
82.66
11.87
5.43
6.23
Table 1: ASR evaluation on lexical and semantic. Results on Canvas Speech and TED-LIUM 3 datasets. MER uses Chinese characters and English words as lexical units. All values are percentages; bold indicates the best result within each model group.
Context Why can a longer vacation feel only slightly better?
Reference There are no new memories added; you have not changed the story.
Hypothesis
WER ↓
SIERlex↓
A: There are no new memories added;
8.33
8.33
you have changed the story.
B: Uh, well, there are no new memories
16.67
0.00
added; you have not changed the story.
Table 2: An example of differing lexical and semantic errors. Hypotheses A and B are produced by Qwen3-ASR-Filetrans and Doubao Filetrans, respectively, for the same utterance from TED-LIUM release 3. A omits the consequential negation “not,” whereas B preserves it but introduces inconsequential fillers. WER favors A, while SIERlex favors B. All values are percentages; lower is better.
Classifier
Acc. ↑
Prec. ↑
Rec. ↑
F1 ↑
Similarity-based metrics
BERTScore [ 26 ]
0.215
0.214
0.910
0.331
SeMaScore [ 20 ]
0.370
0.261
0.910
0.384
LLM-based evaluation
Interactive ASR [ 23 ]
0.515
0.297
0.955
0.440
Direct LLM
0.900
0.764
0.731
0.745
Table 3: Agreement with human semantic consequence judgments. Classification results on discrepant segments from Canvas Speech and TED-LIUM 3, using majority-vote human annotations as ground truth. Consequential is the positive class. Acc., Prec., Rec., and F1 denote accuracy, precision, recall, and F1-score, respectively. All LLM-based methods use DeepSeek-V4-Flash. Bold indicates the highest value in each column.
Figure 2: Target-reference-free correction of SenseVoice on Canvas Speech and TED-LIUM 3. Bars report MER and SIERlex for the original hypothesis, the matched w/o Rubric baseline, and rubric-guided correction. Pink caps indicate newly introduced error mass, while double-headed arrows indicate repaired error mass relative to the original hypothesis.
Evaluation LLM
Acc. ↑
Prec. ↑
Rec. ↑
F1 ↑
GPT-4.1
0.895
0.750
0.800
0.774
GPT-5.5
0.915
0.868
0.733
0.795
GPT-5.6-Luna
0.895
0.714
0.889
0.792
DeepSeek-V4-Flash
0.925
0.857
0.830
0.843
Table 4: Sensitivity of SMART to the evaluation LLM. Semantic consequence classification on the same human-annotated segments using different evaluation LLMs. Consequential is the positive class. Acc., Prec., Rec., and F1 denote accuracy, precision, recall, and F1-score, respectively. Bold indicates the highest value in each column.