From Oracle to Noisy Context: Mitigating Contextual Exposure Bias in Speech-LLMs

Xiaoyong Guo; Nanjie Li; Zijie Zeng; Kai Wang; Hao Huang; Haihua Xu; Wei Shi

cs.CLMar 25, 2026

From Oracle to Noisy Context: Mitigating Contextual Exposure Bias in Speech-LLMs

Authors: Xiaoyong Guo, Nanjie Li, Zijie Zeng, Kai Wang, Hao Huang, Haihua Xu, Wei Shi

Organizations: School of Computer Science and Technology, Xinjiang University, Urumqi, China · Timekettle · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Urumqi, China

Abstract

Contextual automatic speech recognition (ASR) with Speech-LLMs is typically trained with oracle conversation history, but relies on error-prone history at inference, causing a train-test mismatch in the context channel that we term contextual exposure bias. We propose a unified training framework to improve robustness under realistic histories: (i) Teacher Error Knowledge by using Whisper large-v3 hypotheses as training-time history, (ii) Context Dropout to regularize over-reliance on history, and (iii) Direct Preference Optimization (DPO) on curated failure cases. Experiments on TED-LIUM 3 (in-domain) and zero-shot LibriSpeech (out-of-domain) show consistent gains under predicted-history decoding. With a two-utterance history as context, SFT with Whisper hypotheses reduce WER from 5.59% (oracle-history training) to 5.47%, and DPO further improves to 5.17%. Under irrelevant-context attacks, DPO yields the smallest degradation (5.17% -> 5.63%), indicating improved robustness to misleading context. Our code and models are published on https://github.com/XYGuo1996/Contextual_Speech_LLMs.

Figures & tables

$N$	$\text{Con}_{\text{inf}}$ / $\text{Con}_{\text{train}}$	0 Dropout WER (%) $\downarrow$				0.5 Dropout WER (%) $\downarrow$
$N$	$\text{Con}_{\text{inf}}$ / $\text{Con}_{\text{train}}$	TED	Test-clean	Test-other	LS-Ave.	TED	Test-clean	Test-other	LS-Ave.
0	- / -	7.89	4.79	9.83	7.310	-	-	-	-
1	GT / GT	5.6	4.49	10.36	7.425	7.89	4.31	9.68	6.995
	hyp / GT	5.85	4.54	10.63	7.585	7.47	4.74	9.94	7.340
	hyp / Whisper	5.62	4.67	9.46	7.065	7.21	5.37	9.96	7.665
	+ DPO	5.69	4.71	9.57	7.140	5.32	4.56	9.38	6.970

Table 1: WER comparison on TED-LIUM 3 and out-of-domain Librispeech dataset across different context window sizes (

N

). The column

\text{Con}_{\text{inf}}

\text{Con}_{\text{train}}

specifies the source of history used during inference and training, respectively. hyp denotes using the model’s own predictions as history during inference. Regarding training configuration, GT uses ground-truth history, while Whisper indicates the model was trained using context decoded by Whisper to simulate historical errors. + DPO and + SFT2 are additional fine-tuning stages applied to the SFT model.

$N$	$\text{Con}_{\text{inf}}$ / $\text{Con}_{\text{train}}$	0 Dropout WER (%) $\downarrow$			0.5 Dropout WER (%) $\downarrow$
$N$	$\text{Con}_{\text{inf}}$ / $\text{Con}_{\text{train}}$	Attacks/o	Attacks/w	Gap $\downarrow$	Attacks/o	Attacks/w	Gap $\downarrow$
1	hyp / GT	5.85	8.32	2.47	7.47	8.82	1.35
	hyp / Whisper	5.62	6.27	0.65	7.21	7.09	-0.12
	+ DPO	5.69	5.5	-0.19	5.32	5.23	-0.09
2	hyp / GT	6.89	8.43	1.54	5.59	9.23	3.46
	hyp / Whisper	8.15	10.37	2.22	5.47	7.93	2.46

Table 3: Robustness analysis against irrelevant context attacks on TED-LIUM 3. We evaluate model resilience by replacing the historical context with randomly sampled irrelevant context.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

WER (%) Threshold	$\gamma$	TED-LIUM 3 (WER %) $\downarrow$			LibriSpeech (WER %) $\downarrow$
WER (%) Threshold	$\gamma$	Attacks/o	Attacks/w	Gap $\downarrow$	Test-clean	Test-other	Ave.
5	0	5.47	7.93	2.46	5.14	9.50	7.320
	0.0625	5.40	7.02	1.62	5.11	9.32	7.215
	0.125	5.43	7.34	1.91	5.18	9.41	7.295
	0.1875	5.04	6.76	1.72	5.09	9.70	7.395
	0.25	5.23	5.99	0.76	5.18	9.51	7.345

Appendix

Table 4: Impact of Hard Negatives threshold and DPO LoRA scaling factor (

\gamma

) during inference. TED-LIUM 3 Gap denotes the WER degradation caused by irrelevant context attacks.

Contextual ASR

See all 19 papers

How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMs
Aug 6, 2026Christian Huber, Alexander WaibelASR Evaluation Automatic Speech Recognition
Interactive Systems Lab · Karlsruhe Institute of Technology · Karlsruhe, Germany +2
LOGIC: Efficient and Robust Contextual Biasing for Speech LLMs via Logit-Space Integration
Jan 21, 2026Peidong Wang, Jian Xue, Jinyu LiLanguage Model Decoding Speech Processing
Microsoft, USA
Training-Free Contextual ASR via SpeechLLM-Based Error-Aware Selective Retrieval
Sep 25, 2026Natsuo Yamashita, Ai Nemoto, Ryosuke Koichi +1Automatic Speech Recognition
Hitachi, Ltd., Research & Development Group, Japan · Hitachi Advanced Systems Corporation, Japan

Aug 6, 2026J/K move · Enter open · S save

How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMs

Christian Huber, Alexander Waibel

Interactive Systems Lab · Karlsruhe Institute of Technology · Karlsruhe, Germany+2

ASR Evaluation Automatic Speech Recognition Speech Language Models Contextual ASR

Open paper AskPDF arXiv

Recognizing new and rare words - named entities, acronyms, domain specific special words, and other items scarce in training data - remains a key challenge for automatic speech recognition (ASR). We compare two strategies for this: context biasing methods, where an ASR model is extended such that during inference a word list can be supplied, and speech large language models (LLMs) prompted with context directly. We evaluate two context biasing methods based on Whisper against three speech LLMs across read and non-read speech, reporting biased, unbiased, and overall word error rate (WER). The context biasing methods cut biased WER by up to 88% relative while leaving other words largely unaffected. Speech LLMs excel on read speech but generalize less well to non-read speech, and prove sensitive to distractor count and prompt word order. We characterize the resulting trade-offs to guide method selection.

Abstract

Contextual ASR

Explore similar work

How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMs

LOGIC: Efficient and Robust Contextual Biasing for Speech LLMs via Logit-Space Integration

Training-Free Contextual ASR via SpeechLLM-Based Error-Aware Selective Retrieval