Measuring Local Acoustic Contribution in Speech LLMs via Word-Span Counterfactual Interventions

Hardik B. Sailor; Qiongqiong Wang; Dheemant Rastogi; Nancy F. Chen; Ai Ti Aw

cs.SDOct 21, 2026

Measuring Local Acoustic Contribution in Speech LLMs via Word-Span Counterfactual Interventions

Authors: Hardik B. Sailor, Qiongqiong Wang, Dheemant Rastogi, Nancy F. Chen, Ai Ti Aw

Organizations: †Institute of Advanced Intelligence and Computing, Agency for Science, Technology and Research, Singapore · ⋆Nanyang Technological University, Singapore

Abstract

Correct transcription does not establish that a word depended on its local audio. We introduce the Acoustic Evidence Contribution (AEC), a word-level counterfactual measure of acoustic grounding. It is obtained from the change in reference-token log likelihood when the word's aligned acoustic span is replaced. We combine decoding outcomes on unedited audio with thresholded AEC in an outcome-contribution matrix, where correct targets are categorized according to their local acoustic contribution as high or low. Depending on the system, 33-49% of correct single-word targets fall below the twofold contribution threshold when averaged across corpora, despite no evident rank association with WER. Retention of reference words after span removal decreases monotonically across AEC quintiles, from 76-83% in the lowest quintile to approximately 1% in the highest. AEC therefore separates transcription correctness from dependence on the aligned local acoustic span while remaining an oracle-aligned diagnostic method.

Figures & tables

Fig. 1: Three-part analysis for each aligned reference word or phrase span.

Fig. 2: Spatial localization of AEC responses around the removed target span. LibriSpeech exhibits a similar localization pattern and is omitted for space.

Score	Edit info	Ref.-spec.	Corpus (T/A/L)	Edit (S/N/M)
Prior surprisal	None	$\checkmark$	$.58/.55/.54$	$.56/.56/.56$
Entropy $H$	Edit only	$\times$	$.89/.82/.88$	$.85/.87/.87$
$\Delta H$	Paired	$\times$	$.91/.85/\mathbf{.94}$	$.89/.90/.90$
AEC	Paired	$\checkmark$	$\mathbf{.93}/\mathbf{.89}/.90$	$.90/.91/.91$

TABLE I:

\operatorname{AUROC}_{\mathrm{span}}

avg. over the six systems.

Single-word targets
System	TED-LIUM	AMI	LibriSpeech
Whisper-v3 (1.5B)	$36.5\pm 3.6$	$37.5\pm 1.7$	$35.2\pm 2.2$
Qwen3-ASR (1.7B)	$39.1\pm 3.6$	$28.9\pm 1.6$	$33.3\pm 2.0$
Granite Speech 4.1 (2.3B)	$47.3\pm 3.7$	$43.0\pm 1.7$	$40.9\pm 2.3$
MERaLiON-3-ASR (3.3B)	$57.1\pm 3.5$	$43.4\pm 1.8$	$45.7\pm 2.2$
Audio Flamingo Next (8.3B)	$33.0\pm 3.4$	$33.9\pm 1.6$	$32.5\pm 2.2$

TABLE II: % LCCS for single word and span-macro targets.

		LCCS (%)		WER (%)
System	Par.	T/A/L	mean	T/A/L	mean
Audio Flamingo Next	8.3	33.0/33.9/32.5	33.1	3.9/11.8/3.8	6.5
Qwen3-ASR	1.7	39.1/28.9/33.3	33.8	4.4/13.3/4.0	7.2
Whisper-v3	1.5	36.5/37.5/35.2	36.4	6.0/18.3/5.0	9.8
Voxtral Small	24.3	34.9/36.7/40.9	37.5	5.5/17.5/4.4	9.1
Granite Speech 4.1	2.3	47.3/43.0/40.9	43.7	5.2/7.8/2.7	5.2

TABLE III: LCCS on single-word targets and unedited speech WER, per corpus and averaged over the three.

Fig. 3: Retention after evidence removal as a graded function of AEC.

Fig. 4: Calibration of the fixed AEF threshold. Thin lines represent individual systems and thick lines represent corpus means.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 205 papers

Text-Only LLM Correctors Break Correct Transcripts: A Repair-Damage Analysis of Audio Grounding in ASR Error Correction
Oct 21, 2026Vasiliy Kudryavtsev, Kirill Borodin, Nikita Vasiliev +2LLM Self-Correction Automatic Speech Recognition
BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia · lab260, Yerevan, Armenia
Clean Accuracy Does Not Guarantee Provenance Robustness: A Prospective Codec-Stress Evaluation of Audio Attribution
Sep 7, 2026Gang ShiSpeech Foundation Models
Independent Researcher
Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors
Oct 17, 2026Sankalp Parashar, Preethi Jyothi, Sunita SarawagiAutomatic Speech Recognition Streaming ASR
IIT Bombay

Oct 21, 2026J/K move · Enter open · S save

Text-Only LLM Correctors Break Correct Transcripts: A Repair-Damage Analysis of Audio Grounding in ASR Error Correction

Vasiliy Kudryavtsev, Kirill Borodin, Nikita Vasiliev, Mikhail Gorodnichev+1

BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia · lab260, Yerevan, Armenia

LLM Self-Correction Automatic Speech Recognition ASR Evaluation

Open paper AskPDF arXiv

LLM correctors of ASR output are judged by aggregate WER, which hides what they fix and what they break. We decompose every WER change exactly into repaired tokens, damaged tokens and insertions, and use this to ask what the source audio adds to a corrector. We compare correction with and without the audio, for the model's own draft and for the output of an independent recogniser, on three corpora. With Gemini 2.5 Flash, text-only correction of its own draft breaks 13-32% of already-correct transcripts; with the audio, 1-3%. Damage falls in both settings. Repair falls against the own draft and rises against the recogniser. On medical dictation, kept insertions cancel the gain in net WER. The open-weight Voxtral, which rarely edits, does not show this pattern. Prompts differ between the two arms, so the contrasts bound the effect of the audio and do not isolate it.

Abstract

Audio-Language Model Evaluation

Explore similar work

Text-Only LLM Correctors Break Correct Transcripts: A Repair-Damage Analysis of Audio Grounding in ASR Error Correction

Clean Accuracy Does Not Guarantee Provenance Robustness: A Prospective Codec-Stress Evaluation of Audio Attribution

Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors