Select or Rewrite: Oracle-Guided Fine-Tuning for Generative ASR Post-Editing from N-Best Hypotheses

Shiran Liu; Gaohui Xu; Sergio Burdisso; Thibault Bañeras-Roux; Shashi Kumar; Petr Motlicek

cs.CLOct 10, 2026

Select or Rewrite: Oracle-Guided Fine-Tuning for Generative ASR Post-Editing from N-Best Hypotheses

Authors: Shiran Liu, Gaohui Xu, Sergio Burdisso, Thibault Bañeras-Roux, Shashi Kumar, Petr Motlicek

Organizations: Idiap Research Institute, Martigny, Switzerland · KTH Royal Institute of Technology, Stockholm, Sweden · Brno University of Technology, Brno, Czech Republic

Abstract

Large language models can post-edit ASR by using linguistic context to compare N-best hypotheses and rewrite the transcript when needed, often improving WER without retraining the recognizer. However, prompt-only correction is brittle: it may over-edit when top-1 is correct and hallucinate when the reference is absent, and often needs very large models. We propose oracle-guided fine-tuning that turns N-best lists into supervision via three curated pools: corrections (reference in the list but not top-1) to teach selection, regularizers (top-1 correct) to discourage unnecessary rewrites, and hard negatives (reference absent) to suppress hallucination. Fine-tuning on this data provides more effective supervision than random subset fine-tuning for learning a hybrid select-or-rewrite policy, with the clearest gains from around 3B parameters on the proprietary dataset but less conclusive results on CommonVoice, a divergence suggesting that public read-speech text may overlap with LLM pretraining and mask genuine post-editing gains.

Figures & tables

Fig. 1: Pipeline overview of the oracle-guided select-or-rewrite framework. ASR beam search produces N-best lists for training and test utterances. Training N-best lists are compared with references and curated into behavior-targeted pools for LoRA fine-tuning, while test N-best lists are given to the LLM for generative selection or rewriting and evaluated with WER.

	definedAI		CommonVoice
	Train	Test	Train	Test
# Utterances	17398	2009	17000	16354
Total Duration (hrs)	38.17	4.35	27.98	27.05

TABLE I: Statistics of the two datasets used in this work.

	Before Filtering							After Filtering
Category (Training Set)	DAI W-S	DAI W-M	DAI W-L	CV W-S	CV W-M	CV W-L	$\rightarrow$	DAI W-S	DAI W-M	DAI W-L	CV W-S	CV W-M	CV W-L
Positive (Recoverable)	2485	2512	2413	2897	2772	2694	$\rightarrow$	2286	2311	2221	2665	2550	2479
Regularizers (Correct)	5994	6387	6419	9319	10447	10723	$\rightarrow$	1102	1175	1180	1714	1923	1972
Hard Negatives (Hopeless)	8919	8499	8566	4784	3781	3583	$\rightarrow$	686	693	665	800	764	743
Validation (8%)							$\rightarrow$	354	363	353	450	455	452
Total	17398	17398	17398	17000	17000	17000	$\rightarrow$	4074	4179	4066	5179	5237	5194

TABLE II: Statistics of the Oracle-Guided Fine-tuning Dataset . Recoverable errors (

y_{\text{ref}}\in\mathcal{H}

and

y_{\text{top1}}\neq y_{\text{ref}}

) are fully retained. Correct predictions (regularizers) are downsampled by keeping the top 20% according to a risk-weighted ranking score. Additional hard negatives (

y_{\text{ref}}\notin\mathcal{H}

) are sampled at a ratio of 0.3 relative to the number of recoverable errors, filtered by requiring the top-1 WER to lie in

[0.05,0.35]

and the N-best diversity score is set to 0.08. Finally, 8% of the curated samples are reserved as validation using a stratified split over sample category and difficulty bins.

Fig. 2: Comparison between oracle-guided and random-data fine-tuning under a matched training-data budget. The random baseline follows the same fine-tuning setup and uses the same number of examples as the oracle-guided setting, but samples them randomly from the training split without oracle-based curation. Results are shown for Whisper-Small, Whisper-Medium, and Whisper-Large on definedAI and CommonVoice .

Fig. 3: Comparison of zero-shot prompting, few-shot prompting, and oracle-guided fine-tuning for LLM-based ASR post-editing. Results are shown across Qwen2.5 model sizes using Whisper-Small, Whisper-Medium, and Whisper-Large ASR frontends on definedAI and CommonVoice .

Panel of Figure 6 — Fig. 4: Example of LLM-based hypothesis selection using raw ASR hypotheses. The Top-1 hypothesis contains a lexical substitution, where “thanks a lot for” is misrecognized as “things are up with”. The correct phrase appears in lower-ranked N-best candidates, and the LLM selects it to produce the normalized output. This illustrates recoverable-error correction through listwise comparison of raw beam-search hypotheses.

Language Model Post-Training

Language model post-training adapts pretrained models to follow instructions, reason, use tools, and meet task-specific or human-defined goals.

See all 182 papers

Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions
Aug 31, 2026Chan-Jan Hsu, Jaeyeon Kim, Chao-Han Huck Yang +3Automatic Speech Recognition Speech Language Models
Carnegie Mellon University · NVIDIA · Work done at NVIDIA +1
Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis
Sep 21, 2026Chee-En Yu, Yi-Cheng Lin, Sung-Feng Huang +4TTS Synthesis Audio Reasoning
Graduate Institute of Electrical Engineering, National Taiwan University, Taiwan · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · NVIDIA Research 4 Artificial Intelligence Center of Research Excellence, National Taiwan University, Taiwan
Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction
Aug 29, 2026Zeyang Song, Tianchi Liu, Tianrui Wang +3TTS Synthesis Text-to-Speech
National University of Singapore · Tencent · LIGHTSPEED +4

Aug 31, 2026J/K move · Enter open · S save

Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions

Chan-Jan Hsu, Jaeyeon Kim, Chao-Han Huck Yang, Shinji Watanabe+2

Carnegie Mellon University · NVIDIA · Work done at NVIDIA+1

Automatic Speech Recognition Speech Language Models Audio-Language Models LLM Self-Correction

Open paper AskPDF arXiv

Recent automatic speech recognition (ASR) systems increasingly integrate large language models (LLMs) to leverage their semantic knowledge, either externally through logit fusion or internally through warm initialization. However, how to effectively combine these two strategies remains underexplored. In this work, we refine warm-initialized LLM-based ASR models by leveraging their own pre-adaptation base LLMs, focusing on LoRA-adapted settings where the base LLM is preserved. To achieve this, we propose Hybrid Search, a targeted correction strategy motivated by two observations. First, interaction features that characterize the relationship between LLM-based ASR hidden states and base-LLM hidden states provide informative signals about a token's degree of semantic dependence. Second, selectively refining targeted tokens with high semantic dependence improves ASR performance far beyond naive global LLM-correction methods including rescoring and late fusion. Our analysis suggests that, even after semantic knowledge transfer through warm initialization, LLM-based ASR models can still leverage their base LLM to further improve inference-time performance.

Abstract

Language Model Post-Training

Explore similar work

Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions

Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis

Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction