cs.CLAug 28, 2026

No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus

Authors: Theodore O. Cochran, Stephanie Dodson, Keith Nore

Organizations: AI for Altruism · Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Supplying context at inference time to a large multimodal model is an inexpensive lever for adapting speech transcription to a domain, and earlier results on smaller models reported large gains. This work tested that mechanism where it ships, in the prompt-conditioning layer of a production oral-history transcription tool, on a sample from its own production corpus. Full prompt-level context did not detectably change side-level word error rate (WER), and none of the four preregistered hypotheses was supported. The design was a within-item paired ablation, preregistered with the analysis code frozen by hash before the confirmatory batch was scored; two disclosed gpt-4o pilot sides had been scored earlier, during scorer development. Nineteen cassette sides, about 10.6 hours of degraded 1970s-80s interview audio, were reprocessed through the production code path under three prompt arms, crossed with two deployed commercial configurations, gpt-4o-transcribe and gemini-2.5-flash, and scored against operator-corrected verbatim references. For gpt-4o-transcribe the median paired difference between the full-context and no-context arms was +0.6 WER points, with a side-resampled interval of [-1.1, +1.0]; the Gemini estimates were too unstable to support a comparable negative inference. A post-hoc rerun found run-to-run pipeline variability larger than the confirmatory differences, so effects of that size cannot be resolved from one transcription per cell. An implementation audit verified the manipulation was live, and sequence-alignment analysis found a small improvement on complete context-listed phrases, too small to materially change side-level WER, and for Gemini coexisting with worsened unlisted-token error. Evaluating context mechanisms therefore requires sequence-aligned term-level, insertion, and speaker-label measures alongside aggregate accuracy.

Figures & tables

Figure 1: Both vertical axes are inverted, so improvement points upward, and color, marker, and line style encode it as well. Top: all 19 sides’ paired WER, no prompt →\rightarrow full context, per deployed configuration (log scale; shaded band == pathological, WER >> 100%). Bottom: the same paired differences (full −- none), sorted, on a linear point scale, which is the direct estimand. gpt-4o-transcribe changes were mixed and centered near zero (12 worse / 7 better; median Δ\Delta +0.6+0.6 points; median ∣Δ∣|\Delta| 1.4; signed- Δ\Delta IQR [−1.3,+1.5][-1.3,+1.5] ), although several sides showed larger individual changes (signed- Δ\Delta range −21.1-21.1 to +11.0+11.0 ). gemini-2.5-flash is dominated by arm-unstable high-WER outputs (median ∣Δ∣|\Delta| 7.0, range −269.9-269.9 to +72.7+72.7 ): three of its five no-prompt-pathological sides recover under full context, one improves but remains pathological, one worsens within the band, and one previously healthy side crosses in (the arm-instability and corpus-weighted divergence reported above).
More in this topic

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 484 papers

Explore similar work

CardsList
Oct 19, 2026

Think Before You Transcribe: Context-Steered Reasoning for Hotword-Aware Speech Recognition

Contextual automatic speech recognition (ASR) improves rare-entity recognition by supplying candidate hotwords, but retrieved candidates may be incomplete or incorrect. A reliable system must therefore exploit a relevant candidate without allowing unsupported alternatives to override the acoustic evidence. We propose a context-steered reasoning framework that generates a short, audio-grounded caption before transcription. This task-specific representation lets the model assess candidate relevance before making a lexical decision. Supervised fine-tuning establishes the structured caption-to-transcription behavior, while targeted trajectory injection strengthens group relative policy optimization when difficult utterances yield only unsuccessful rollouts. Experiments across diverse context conditions show that the proposed method consistently improves both transcription quality and hotword recognition while remaining robust to irrelevant candidates.

Qihao Wang, Hang Su, Fan Cui +3

Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · MiLM Plus, Xiaomi Inc.

Automatic Speech RecognitionContextual ASRRobust ASRAudio Reasoning
AskPDF
Jul 24, 2026

Listen, Do Not Copy: Internalizing Audio-Grounded Scaffold Context for Robust Omni-Model Speech Understanding

Omni models transcribe clean, single-speaker speech well, but their accuracy drops sharply when speakers overlap and the scene is noisy, exactly where knowing who said what matters most. A natural fix is a short scene description. We show why this is risky: answer-bearing text lets the model copy instead of listen, so the score rises although nothing has been heard; a silent test exposes this shortcut at once. We call this failure mode perception bypass and address it with Audio-Grounded Scaffold Context (AGSC). AGSC links three steps: first, we build clues from audio to guide listening without giving the answer; second, answer-overlap and silence tests probe them for leakage and audio dependence; finally, those clues scaffold training but vanish at test time, yielding no-clue capability. Across three heterogeneous Omni models, training on AGSC lowers no-clue capped mean permutation word error rate (mpWER) on overlapping, noisy speech from 25%-71% to 9%-15%. For streaming control, we formulate a joint GDPO task in which the model learns when to use a clue and how to produce a speaker-attributed transcript from separately normalized format, gate, and transcript rewards. After internalization, AGSC adds almost no inference overhead.

Pengfei Zhang, Biao Tian, Tianxin Xie +3

AI Thrust, Information Hub, The Hong Kong University of Science and Technology (Guangzhou) · Token Foundry, Alibaba Group, China

Speaker-Attributed ASRContextual ASRStreaming ASRAutomatic Speech RecognitionShortcut LearningSpeaker Diarization
AskPDF
Mar 25, 2026

From Oracle to Noisy Context: Mitigating Contextual Exposure Bias in Speech-LLMs

Contextual automatic speech recognition (ASR) with Speech-LLMs is typically trained with oracle conversation history, but relies on error-prone history at inference, causing a train-test mismatch in the context channel that we term contextual exposure bias. We propose a unified training framework to improve robustness under realistic histories: (i) Teacher Error Knowledge by using Whisper large-v3 hypotheses as training-time history, (ii) Context Dropout to regularize over-reliance on history, and (iii) Direct Preference Optimization (DPO) on curated failure cases. Experiments on TED-LIUM 3 (in-domain) and zero-shot LibriSpeech (out-of-domain) show consistent gains under predicted-history decoding. With a two-utterance history as context, SFT with Whisper hypotheses reduce WER from 5.59% (oracle-history training) to 5.47%, and DPO further improves to 5.17%. Under irrelevant-context attacks, DPO yields the smallest degradation (5.17% -> 5.63%), indicating improved robustness to misleading context. Our code and models are published on https://github.com/XYGuo1996/Contextual_Speech_LLMs.

Xiaoyong Guo, Nanjie Li, Zijie Zeng +4

School of Computer Science and Technology, Xinjiang University, Urumqi, China · Timekettle · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Urumqi, China

Contextual ASRAutomatic Speech RecognitionSpeech Language ModelsLanguage Model Robustness
AskPDF