From Transcription to Reformulation: Defining and Benchmarking Grounded Generative Speech Recognition

Qing Wang; Qihan Hu; Hang Lv; Yan Yang; Boqing Liu; Xuechao Zhu; Zehan Li; Hongjie Chen

cs.CLOct 30, 2026

From Transcription to Reformulation: Defining and Benchmarking Grounded Generative Speech Recognition

Authors: Qing Wang, Qihan Hu, Hang Lv, Yan Yang, Boqing Liu, Xuechao Zhu, Zehan Li, Hongjie Chen,

Organizations: Xingchen AGI Lab, China Telecom Artificial Intelligence Technology Co. Ltd, China

Abstract

Speech language models have expanded automatic speech recognition (ASR) from transcription to general speech-to-text generation. Yet existing benchmarks still assume a single ground-truth transcript per utterance, an assumption that struggles when models produce readable, formatted, or structured outputs. In this work, we introduce Grounded Generative Speech Recognition (GSR), reformulating speech recognition as generating semantically grounded text instead of a single transcript. We define a grounded reformulation space that characterizes all valid textual realizations supported by the input speech, and establish a four-level hierarchy of reformulation capabilities, including literal transcription, canonical reformulation, semantic reformulation and speech-grounded structuring. Based on this formulation, we present GSR-Bench, the first comprehensive benchmark designed to evaluate generative speech recognition systems beyond exact transcript matching. GSR-Bench covers diverse scenarios including refined meeting transcription, cross-dialect translation, context-aware speech understanding, and robustness under silent or noisy acoustic conditions. To address the limitations of word-level evaluation metrics, we develop a multi-dimensional evaluation framework combining speech grounding assessment and LLM-based semantic judgment, whose reliability is validated through human preference studies. Extensive experiments across representative architectures, ranging from conventional ASR systems to end-to-end Speech-LLMs, reveal fundamental trade-offs between transcription fidelity and generative reformulation ability. Our analysis further identifies critical failure modes, including acoustic hallucination, unsupported generation, and instruction drift, providing insights into the design and evaluation of future grounded generative speech systems.

Figures & tables

Figure 1: Overview of GSR-Bench: Data Construction, Quality Assurance, and Hierarchical Task Organization.

Figure 2: GSR-Bench Evaluation Framework: Reformulation Hierarchy, System Coverage, and Grounding Reliability.

System	GPT-4o			DeepSeek-v4			LLM-Score $\uparrow$	CER $\downarrow$
System	Fidelity	Fluency	Readability	Fidelity	Fluency	Readability	LLM-Score $\uparrow$	CER $\downarrow$
TeleASR ( Chen et al. 2024a )	4.86	4.07	4.02	4.77	3.58	3.66	4.16	3.46
UniGSR-ASR	4.84	4.11	4.05	4.78	3.67	3.74	4.20	3.96
FireRed-LLM ( Xu et al. 2025 )	4.76	3.87	3.80	4.50	4.06	3.94	4.16	4.45
FireRed-AED ( Xu et al. 2025 )	4.73	3.86	3.78	4.47	4.05	3.91	4.13	6.06
FunASR-Nano ( An et al. 2025 )	4.65	3.85	3.74	4.62	3.71	3.69	4.04	6.10

Table 1: Multi-dimensional LLM-as-Judge evaluation on GSR-Refine. LLM-Scores range from 1 to 5; higher is better.

Context	FunASR	Qwen3-ASR-1.7B	UniGSR
No Context	6.35	8.73	8.52
Hotwords	4.30	7.97	4.79
Domain	6.20	8.86	9.65
Long Context	5.87	6.26	5.73
Dialogue	3.76	6.31	6.01
User Profile	6.18	8.85	17.00

Table 2: Context-conditioned transcription results (CER %, lower is better) under different context configurations. Bold marks the best result per system. Values below the No-Ctx baseline indicate successful context utilization; values above indicate degradation.

System	Dialect	BLEU	COMET	BERTScore	Sent-BERT	LLM (4o)	LLM (v4)
TeleASR $\rightarrow$ LLM (Cascade)	Yue	0.330	0.873	0.884	0.892	3.32	3.19
	Wu	0.532	0.914	0.934	0.932	3.89	3.62
	SW Mandarin	0.526	0.833	0.901	0.882	3.2	3.25
	Henan	0.307	0.809	0.860	0.859	2.98	3.08
UniGSR	Yue	0.297	0.849	0.847	0.872	3.61	3.36
	Wu	0.366	0.895	0.905	0.911	4.25	4.03

Table 3: Evaluation of Level-2 cross-dialect semantic reformulation. BLEU, COMET, BERTScore, and Sentence-BERT measure lexical or embedding similarity against a single reference, whereas GPT-4o and DeepSeek-v4 evaluate semantic quality and translation naturalness (1–5, higher is better).

Metric	Gemini 3.1 Pro	StepAudio-R1	UniGSR
Faithfulness	4.41	3.79	3.14
Coverage	4.85	4.58	4.21
Key Information	4.66	4.18	3.53
Conciseness	4.95	4.84	4.74
Fluency	4.97	4.87	4.60

Table 4: LLM-as-Judge evaluation for Level-3 speech-grounded summarization. Five quality dimensions are scored from 1 to 5, where higher scores indicate better summary quality.

System	HTR (%) $\downarrow$	AHL $\downarrow$	Total Chars $\downarrow$
UniGSR	15.0	17.28	1348
FunASR-Nano	6.9	54.08	1947
Qwen-ASR-0.6B	14.4	11.33	850
Qwen-ASR-1.7B	9.8	10.31	526
FireRed	16.0	14.16	1175
Gemini3.1 Pro	10.8	12.57	704

Table 5: Grounding reliability under acoustic stress. The evaluation suite contains 520 audio segments spanning environmental noise, silence, music, and unintelligible weak speech. Any non-empty output is counted as an acoustic hallucination. Lower is better.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 493 papers

Jul 30, 2026

AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach

Automatic speech recognition (ASR) has achieved substantial gains in transcription accuracy, yet verbatim transcription does not necessarily produce readily usable text. It retains fillers, repetitions, false starts, and self-corrections that increase reading effort, obscure the speaker's final intent, and propagate unresolved or abandoned content to downstream tasks. Existing spoken-to-written methods process completed audio or transcripts but cannot revise emitted text when later speech changes how preceding content should be interpreted. We therefore formulate Agentic Speech Recognition (AgenticSR), an audio-to-clean-text task that removes disfluencies, resolves self-corrections, and normalizes written form while preserving the speaker's final intent. AgenticASR implements this task through an ASR--Refiner architecture that repeatedly transforms a bounded active context and replaces its corresponding output span as audio arrives. This enables continual emission and revision over streams of arbitrary duration. We also introduce AASR-Bench, a bilingual benchmark with fine-grained atomic rubrics. Across multiple ASR front ends, AgenticASR attains the highest AASR-Bench scores among evaluated systems. A human--AI agreement study shows that rubric-based judgments align with independent expert assessments. Ablations characterize Refiner capacity, context length, and the quality--latency trade-off between online and offline inference. Together, these results establish AgenticASR as a practical framework for intent-preserving clean transcription during ongoing speech. Code, AASR-Bench, and a demo will be released at https://github.com/AnXMuy/AgenticASR.

Zixuan Jiang, Binghao Qiang, Jiaying Chi +3

X-LANCE lab, Shanghai Jiao Tong University · Shanghai Innovation Institute · College of Artificial Intelligence, Xi’an Jiaotong University

ASR Evaluation Streaming ASR Speech Processing Automatic Speech Recognition

Ask PDF

Oct 22, 2026

DRAMA-Eval: A Benchmark for Context-Aware, Multi-Turn, Multi-Speaker Speech Generation

Recent advances in speech generation are shifting toward context-aware, multi-turn, and multi-speaker synthesis. While these capabilities are in high demand for applications such as audiobook and audio drama production, evaluation has not kept pace with these advances. Beyond traditional zero-shot Text-to-Speech (TTS), modern systems in this domain primarily fall into three categories: instruction-controllable generation models, multi-turn dialogue models, and the audio branches of audio-visual generation models. However, existing benchmarks evaluate these paradigms in isolation, failing to provide a unified assessment of context modeling, instruction following, and multi-speaker generation. Meanwhile, recent progress in audio understanding models has made it feasible to evaluate such complex scenarios using LLM-as-a-Judge approaches. To bridge this gap, we introduce DRAMA-Eval, a comprehensive benchmark for evaluating such systems. DRAMA-Eval utilizes structured script contexts and offers a holistic evaluation framework covering objective, local, and global aspects of generation quality, enabling the assessment of acoustic fidelity, sentence-level execution, and long-range coherence. We benchmark leading open-source and commercial models on a human?verified dataset. The results reveal critical weaknesses in long-range speaker consistency and implicit instruction following that require deep contextual understanding. To facilitate reproducible research, we release a 12-hour bilingual (Chinese-English) test set of manually corrected real-world scripts, accompanied by ground-truth audio, evaluation code, and benchmark results.

Jingbin Hu, Zihang Yang, Xiaoge Peng +21

Shenzhen Pimei Technology Co., Ltd. · Yutu Zhineng

Speech Generation Evaluation Multi-Speaker TTS

Ask PDF

Jul 10, 2026

Generative Testing of Automated Speech Recognition Systems

Automatic speech recognition (ASR) systems have achieved high accuracy with transformer-based models, enabling deployment in critical applications. However, they remain vulnerable to adversarial manipulation, particularly in black-box settings where attacks must preserve perceptual naturalness. This work introduces GATAS, a black-box testing approach that generates failure inducing inputs by operating in the phoneme-level latent space of a text- to-speech model. Instead of perturbing waveforms directly, the approach interpolates latent representations to induce transcription errors while remaining within the manifold of natural speech. The attack is formulated as a multi-objective optimization problem balancing semantic divergence and perceptual quality. Our empirical evaluation against both white-box and black-box baselines shows that GATAS achieves a 98% success rate while producing lower distortion and higher perceptual quality, as confirmed by human studies. Despite operating without gradient access, GATAS remains competitive against white-box methods, highlighting that representation and perceptual alignment are more critical than access to model internals. Overall, our results demonstrate that untargeted latent-space optimization enables the efficient generation of realistic and effective test cases for ASR systems.

Yanis Xabier Wilbrand Peña, Oliver Weißl, Andrea Stocco

Technical University of Munich Germany · Technical University of Munich & fortiss GmbH Germany

ASR Evaluation Adversarial Robustness of ASR Black-Box Adversarial Attacks

Ask PDF

Abstract

Automatic Speech Recognition

Explore similar work

AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach

DRAMA-Eval: A Benchmark for Context-Aware, Multi-Turn, Multi-Speaker Speech Generation

Generative Testing of Automated Speech Recognition Systems