Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition

Peng Wang; Yanqiao Zhu; Zixuan Jiang; Qinyuan Chen; Xingjian Zhao; Xipeng Qiu; Wupeng Wang; Zhifu Gao

cs.CLApr 10, 2026

Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition

Authors: Peng Wang, Yanqiao Zhu, Zixuan Jiang, Qinyuan Chen, Xingjian Zhao, Xipeng Qiu, Wupeng Wang, Zhifu Gao,

Organizations: The Chinese University of Hong Kong, Shenzhen · X-LANCE Lab, Shanghai Jiao Tong University · Xi’an Jiaotong University · Fudan University ·

Abstract

Recent years have witnessed remarkable progress in automatic speech recognition (ASR), driven by advances in model architectures and large-scale training data. However, two important aspects remain underexplored. First, Word Error Rate (WER), the dominant evaluation metric for decades, treats all words equally and often fails to reflect the semantic correctness of an utterance at the sentence level. Second, interactive correction-an essential component of human communication-has rarely been systematically studied in ASR research. In this paper, we integrate these two perspectives under an agentic framework for interactive ASR. We propose leveraging LLM-as-a-Judge as a semantic-aware evaluation metric to assess recognition quality beyond token-level accuracy. Furthermore, we design an LLM-driven agent framework to simulate human-like multi-turn interaction, enabling iterative refinement of recognition outputs through semantic feedback. Extensive experiments are conducted on standard benchmarks, including GigaSpeech (English), WenetSpeech (Chinese), the ASRU 2019 code-switching test set. Both objective and subjective evaluations demonstrate the effectiveness of the proposed framework in improving semantic fidelity and interactive correction capability. We will release the code to facilitate future research in interactive and agentic ASR.

Figures & tables

Figure 1: Traditional vs. Interactive ASR paradigms. (Top) Traditional systems struggle with named entities (e.g., ”Night” vs. ”Knight”); (Bottom), our proposed Interactive ASR can take user’s spoken language instructions (e.g., ”starts with a K”) as the feedback to update and correct ASR results.

Figure 2: Overview of the Interactive ASR framework. An LLM Intent Router classifies the base ASR hypothesis (

H_{t}

) using the previous transcript (

Y_{t-1}

). New utterances bypass correction and are output directly, while corrective instructions trigger an LLM Reasoning Corrector to refine

Y_{t-1}

via a three-step CoT process (Locate, Reason, Surgical Replacement).

Figure 3: Overview of the Automated Simulation Framework. An LLM-as-a-judge first evaluates the semantic coherence of the initial ASR hypothesis. Upon detecting an error, the User Simulator generates a spoken correction instruction via an LLM and TTS. The Interactive ASR then processes this feedback to reason and update the hypothesis, forming an automated correction process.

Dataset	LLM ( $r$ )	Expert ( $r$ )
GigaSpeech	0.8730	0.8345
WenetSpeech	0.7873	0.7351
ASRU2019	0.8556	0.8613
Overall	0.8281	0.8104

Table 1: Pearson correlation coefficients of LLM and Expert judgments against the Human Ground Truth. Bold values indicate the higher correlation in each row.

Loop	GigaSpeech			WenetSpeech			ASRU2019
Loop	WER	SER	$S^{2}ER$	CER	SER	$S^{2}ER$	MER	SER	$S^{2}ER$
0	12.25	61.17	14.12	6.89	35.24	15.56	6.60	38.85	26.89
1	11.08	58.56	6.03	4.59	28.59	6.26	3.59	25.22	8.10
2	10.82	58.03	3.66	4.07	26.97	3.81	3.21	23.04	4.59
3	10.68	57.80	2.67	3.82	26.30	2.71	3.09	22.08	3.06
10	10.53	57.59	1.08	3.51	25.32	1.11	2.88	20.88	0.82

Table 2: Performance evolution of the proposed sentence-level semantic error rate (

S^{2}

ER) versus conventional metrics (WER, CER, MER, SER) across iterative interaction rounds on three datasets.

Figure 4: Comparison of

S^{2}ER

(top) and CER/WER/MER (bottom) reduction trends across three datasets.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 480 papers

Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation
May 28, 2026Zixuan Jiang, Yanqiao Zhu, Peng Wang +8Automatic Speech Recognition
College of Artificial Intelligence, Xi’an Jiaotong University, Xi’an 710049, China · X-LANCE Lab, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China · The Chinese University of Hong Kong, Shenzhen, Shenzhen 518172, China +2
Evaluation of Automatic Speech Recognition Using Generative Large Language Models
Apr 23, 2026Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil +6ASR Evaluation Automatic Speech Recognition
Idiap Research Institute, Martigny, Switzerland · EPFL, Lausanne, Switzerland · Brno University of Technology, Czech Republic +3
AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach
Jul 30, 2026Zixuan Jiang, Binghao Qiang, Jiaying Chi +3ASR Evaluation Streaming ASR
X-LANCE lab, Shanghai Jiao Tong University · Shanghai Innovation Institute · College of Artificial Intelligence, Xi’an Jiaotong University

May 28, 2026J/K move · Enter open · S save

Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation

Zixuan Jiang, Yanqiao Zhu, Peng Wang, Qinyuan Chen+7

College of Artificial Intelligence, Xi’an Jiaotong University, Xi’an 710049, China · X-LANCE Lab, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China · The Chinese University of Hong Kong, Shenzhen, Shenzhen 518172, China+2

Automatic Speech Recognition

Open paper AskPDF arXiv

Automatic speech recognition (ASR) is a core component of human--computer interaction and an increasingly important front-end for LLM-based assistants and agents. However, most current ASR systems still follow a single-pass paradigm, which is poorly aligned with human communication, where misunderstandings are resolved through iterative clarification and refinement. This mismatch makes it difficult to correct meaning-critical errors once they occur. Meanwhile, token-level metrics such as WER or CER cannot adequately reflect such a problem. To address these limitations, we formulate Interactive ASR as a multi-turn refinement task and propose Agentic ASR, a closed-loop framework that combines a single-pass ASR front-end with semantic correction, intent routing, and reasoning-based editing. We further introduce the Sentence-level Semantic Error Rate (

S^2ER

), an LLM-based semantic evaluation metric, together with an Interactive Simulation System for scalable and reproducible benchmarking. Experiments on multilingual, named-entity-intensive, and code-switching benchmarks show that iterative interaction consistently reduces semantic errors, with much larger gains in

S^2ER

than in conventional token-level metrics. Human--AI alignment and ablation studies further validate the reliability of the semantic judge and the robustness of the proposed framework. The code is available at: https://interactiveasr.github.io/ and the live demo is available at https://i-asr.sjtuxlance.com/

Abstract

Automatic Speech Recognition

Explore similar work

Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation

Evaluation of Automatic Speech Recognition Using Generative Large Language Models

AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach