How to Reduce Whisper Hallucination

Husein Zolkepli

cs.CLSep 26, 2026

How to Reduce Whisper Hallucination

Authors: Husein Zolkepli

Organizations: Scicom (MSC) Berhad, Malaysia

Abstract

Whisper is still what runs in production: one permissively licensed checkpoint, 99 languages, no per-language tuning. But it writes sentences nobody said. On 42 clips of pure room tone, whisper-large-v3 emits words on 61.9% of them and emits something on 100%. The usual response is to distil a student from Whisper pseudo-labels, filtering the hallucinations out of the corpus first, which deletes the evidence while leaving the behaviour in place: the student is fitted to the subset where the teacher was right, and inherits a failure mode absent from its own training data. The second response, adding non-speech audio so the model learns to stay quiet, already ships and works, but teaches suppression without discrimination. The checkpoint best on every non-speech arm here is also the one that recovers the fewest genuinely spoken phrases and deletes 58% of repeated speech. The teacher has to be fixed, with both halves of the signal. Our benchmark scores both at once: 11,852 clips over eight arms, 6,267 synthetic positives, 8,296 clips of real audio that made a production model fail, and FLEURS in 58 languages. We collect 40,891 hallucination phrases in 100 languages, choose a text-to-speech system by measurement, and synthesise those phrases as positives, so the model meets the same text both as something to suppress and as something to transcribe. Across 33 matched pairs of fine-tunes differing only by those positives, adding them lowers word emission on real voice-free audio in 31 pairs and raises phrase recovery in 32; among the 30 pairs that hold English accuracy it is 30 out of 30. The best checkpoint takes hallucination on silence from 61.9% to 2.4% and words over real voice-free audio from 99.9% to 47.8%, while raising phrase recovery from 69.8% to 82.7%. The benchmark, lexicon and synthetic corpus are released at https://huggingface.co/datasets/Scicom-intl/Whisper-Hallucination

Figures & tables

LABEL	IDENTIFIER		WHAT IT IS
large-v2	openai/	whisper-large-v2	base, 1.55 B
large-v3	openai/	whisper-large-v3	base, 1.55 B
turbo	openai/	whisper-large-v3-turbo	base, 4 layers
malaysian-v2	mesolitica/	malaysian-whisper-large-v2	Malay tune
malaysian-turbo-v3	mesolitica/	Malaysian-whisper-large-v3-turbo-v3	Malay tune

Table 1: The checkpoints, and the short labels used in every table and figure below. The two Malay fine-tunes are built on large-v2 and turbo respectively, and turbo differs from large-v3 in having four decoder layers rather than thirty-two.

ARM	N	LARGE-V2	LARGE-V3	TURBO	MALAYSIAN -V2	MALAYSIAN -TURBO-V3
silence	42	85.7%	61.9%	59.5%	35.7%	16.7%
music	600	98.7%	97.0%	96.7%	69.7%	31.0%
nonspeech	1,168	98.8%	89.9%	80.7%	76.0%	9.3%
silence any output	42	100%	100%	100%	64.3%	16.7%
music any output	600	100%	100%	100%	73.0%	31.7%
nonspeech any output	1,168	100%	100%	100%	77.8%	9.4%

Table 2: Hallucination on audio that contains no speech. The correct output is the empty string, so any output at all is wrong. The first block counts outputs containing words, the second counts any output including a bare full stop.

ARM	SOURCE	CLIPS	CORRECT OUTPUT
silence	6 noise floors $\times$ 7 durations	42	nothing
music	Free Music Archive ( Defferrard et al., 2017 )	600	nothing
nonspeech	FSD50K ( Fonseca et al., 2020 ) , label-verified voice-free	1,168	nothing
reduplication	a unit repeated an exact number of times	1,440	that many repeats
speech_in_noise	genuine speech mixed at 5 SNRs	1,200	the transcript
genuine	real Malaysian speech	4,694	the transcript

Table 3: The eight benchmark arms. Every arm is a test split. The training corpus is a separate build that is disjoint by construction, verified before any training run.

CORPUS	CHARACTER	KEPT	RATE
AudioSet ( Gemmeke et al., 2017 )	YouTube, heavy background noise	4,376	54.70%
AMI ( Carletta et al., 2005 )	spontaneous meetings, far-field	235	2.94%
GigaSpeech ( Chen et al., 2021a )	podcasts and YouTube, speech-aligned	33	0.41%
Earnings-22 ( Rio et al., 2022 )	conference calls	31	0.39%
People’s Speech dirty ( Galvez et al., 2021 )	noisy transcripts, clean audio	9	0.11%
VoxPopuli ( Wang et al., 2021 )	parliament	4	0.05%

Table 4: Hallucination yield by corpus, per 8,000 clips heard. Recording quality predicts yield across five thousandfold.

Figure 1: Every checkpoint trades one failure for the other. Horizontal axis is how often the model emits words over audio with no speech in it, pooled over the three non-speech arms and weighted by clip count. Vertical axis is how often it recovers a phrase that is genuinely spoken. The top left corner is the model everyone wants and nothing is there.

Figure 2: Hallucination on the three non-speech arms. Every bar is error, because the correct output is the empty string.

Figure 3: Both extremes are wrong on repeated speech. A model that emits more repeats than were spoken is running away; a model that emits nothing has deleted the speech. Malaysian-turbo-v3 does the second on 58.0% of clips.

ARM	N	LARGE-V2	LARGE-V3	TURBO	MALAYSIAN -V2	MALAYSIAN -TURBO-V3
librispeech WER	2,620	4.5%	3.5%	3.5%	3.5%	10.6%
genuine WER	4,694	35.4%	31.8%	31.8%	42.3%	60.7%
speech_in_noise WER	1,200	41.6%	34.5%	36.1%	51.1%	58.7%
fleurs CER, typical language	58 langs	0.097	0.075	0.078	1.051	1.259
fleurs CER, macro mean	1,160	0.422	0.305	0.511	1.781	1.681

Table 5: Accuracy. The last two rows are the multilingual guard, in character error rate.

Figure 4: English accuracy hides what a multilingual fine-tune destroyed. malaysian-whisper-v2 matches base large-v3 on LibriSpeech and returns character error rate 1.05 on the typical FLEURS language.

Figure 5: The same measurement language by language, ordered by base large-v3 difficulty. Whisper genuinely cannot transcribe the right hand end: Amharic, Khmer and Somali sit above 1.4 for every checkpoint. The two Malay fine-tunes sit above the usable line almost everywhere, which is the damage LibriSpeech cannot report.

Figure 6: The ranking inverts when the phrase is really spoken. 6,267 clips in 83 languages, each run three ways. Padding is applied to the same clips, so each triple is a within-clip comparison.

MODEL	RECOVERED, BARE $\to$ TONE	MEDIAN CER	MEAN CER	$>2\times$
whisper-large-v2	63.6% to 63.4%	0.125	0.378 to 0.384	0.6% to 0.7%
whisper-large-v3	69.8% to 68.5%	0.089	0.288 to 0.302	0.2% to 0.4%
whisper-large-v3-turbo	68.0% to 67.0%	0.087	0.348 to 0.382	0.7% to 0.9%
malaysian-whisper-v2	31.3% to 29.6%	0.800	0.770 to 1.151	2.6% to 3.7%
Malaysian-turbo-v3	41.1% to 35.4%	0.385	1.508 to 3.749	3.0% to 5.7%

Table 6: lexicon_synth test split. Emitting the phrase is correct here.

Figure 7: Mean character error rate on the positive arm is a tail statistic. The median barely moves between checkpoints while the mean triples, because a few percent of clips run past twice the phrase length. Report both.

Figure 8: Hallucination yield by corpus, per 8,000 clips heard and mined with large-v3, on a log scale. Curated corpora barely fail. GigaSpeech is the right material and still yields 0.41%, because its segments are cut to speech.

Figure 9: Real audio that a voice activity detector confirms has no speech in it. Every OpenAI checkpoint writes something on all 4,421 clips. Most of what they write is a phrase from the lexicon, which is what makes a blocklist work, but only once the clip is already known to be blank.

SOURCE	WHAT IT IS	PHRASES	LICENCE
AGH DSP full tally ( Barański et al., 2025 )	Whisper output over non-speech audio	30,407	MIT
AGH DSP “Bag of Hallucinations” ( Barański et al., 2025 )	the curated subset of the above	294	MIT
whisper-hallucinations ( Arbonel, 2024 )	community-collected, multilingual	7,889	MIT
NVIDIA NeMo SDP, Granary ( Koluguri et al., 2025 )	the pipeline’s own filter lists, 23 languages	3,028	Apache-2.0

Table 7: Where the lexicon comes from. All four are public and permissively licensed, and all four are vendored in the release so the merge is reproducible.

Figure 10: Text-to-speech candidates on 48 lexicon phrases in 22 languages, identical text, scored by recognition round trip. Coverage and quality are not the same axis. Scicom’s own model is the blue bar.

	MEAN CER	MEDIAN	WINS	LANGUAGES	VOICES
Scicom Multilingual-Expressive-TTS-1.7B	0.221	0.000	16	untagged	45 named
OmniVoice (auto)	0.349	0.018	2	646	1 per language
OmniVoice (voice design)	0.589	0.062	3	646	48 tags
ToucanTTS	0.830	0.586	0	7,233	sampled from a GAN
Higgs Audio v2	1.089	0.713	0	untagged	cloning
Higgs Audio v3	1.230	0.483	1	82/100	cloning

Table 8: Text-to-speech candidates. Round-trip character error rate, lower is better.

Figure 11: One voice per language, and what fixed it, on the 33 OmniVoice languages that sat at 0.75 or worse. Pairwise speaker similarity on a calibrated scale where 0.605 is two strangers and 0.853 is two clips of the same person.

ROUTE	YIELD MK / GU / IT	MEDIAN CER	DIFFERENT-VOICE COSINE
Scicom Multilingual-Expressive, speaker name	86% / 80% / 79%	0.07 / 0.17 / 0.00	0.65 / 0.64 / 0.58
OmniVoice, reference cloning	46% / 68% / 53%	0.32 / 0.25 / 0.26	0.68 / 0.72 / 0.69
auto mode, for reference	89% / 73% / 83%	0.02 / 0.11 / 0.00	0.91 / 0.81 / 0.77

Table 9: Two routes to distinct voices, probed on the same 4 languages, 24 phrases and 4 voices. Lower cosine means better separated voices.

Figure 12: Voice conversion candidates. Character error rate added on top of the source clip against how far the voice actually moved. A converter that returns its input scores a perfect zero cost and is worthless, which is why the vertical axis is here at all.

CONDITIONING	CER	$\Delta$ CER	$\to$ TARGET	DURATION $\times$	$>2\times$
speaker name	0.308	$-$ 0.035	n/a	1.03	2.7%
reference clip, zero shot	1.740	$+$ 1.397	0.861	3.27	67.9%
Higgs v3, for comparison	0.653	$+$ 0.311	0.813	0.90	2.7%

Table 10: The two ways to condition Scicom’s Multilingual-Expressive, measured on the same phrases. Naming a speaker is the best result in the whole comparison. Zero-shot cloning from a reference clip is the worst, and the reason is duration, not timbre.

SOURCE	CLIPS	HOURS	TARGET	TEACHES
corpus nonspeech (FSD50K dev)	16,983	33.3	empty	noise means say nothing
corpus music (FMA shards 2-12)	7,170	58.3	empty	music means say nothing
corpus Malay speech, Emilia ( He et al., 2024 )	5,801	14.8	transcript	keep accuracy
corpus reduplication	2,514	4.0	exact repeats	do not run away
corpus silence	59	0.7	empty	room tone means say nothing
lexicon_synth train	22,845	12.5	the phrase	the phrase is real, transcribe it

Table 11: Training sets. Everything is disjoint from the benchmark by construction and every test split stays held out.

MIX	CLIPS	BLANK SHARE	CONTAINS lexicon_synth
no_synth	36,183	77%	no
all	59,028	47%	yes, 22,845 clips

Table 12: The paired mixes. no_synth is all with lexicon_synth removed and nothing else changed.

SETTING	VALUE
optimiser	AdamW, fused
$\beta_{1}$ , $\beta_{2}$ , $\epsilon$	0.9, 0.999, $1\times 10^{-8}$
weight decay	0.0
schedule	linear decay, 50 warmup steps
gradient clipping	1.0
precision	bf16

Table 13: Training configuration, identical for every one of the 66 runs except the three factors being swept: method, rank and learning rate. One GPU per run, so the per-device batch is the batch.

FAMILY	MODULES	$r=8$	$r=16$	$r=32$	$r=64$	$r=128$
attention only	q_proj, k_proj, v_proj, out_proj	7.9 M	15.7 M	31.5 M	62.9 M	125.8 M
		0.51%	1.01%	2.00%	3.92%	7.54%
attention + MLP	the four above, plus fc1, fc2	14.4 M	28.8 M	57.7 M	115.3 M	230.7 M
		0.93%	1.83%	3.60%	6.95%	13.00%

Table 14: The two adapter families. Trainable counts are measured from the runs, not estimated, and are exactly linear in rank. At every rank, adding the MLP pair costs 1.83 times the attention-only count.

METHOD	RANKS ( $\alpha=2r$ )	LEARNING RATES
LoRA, attention only	8, 16, 32, 64, 128	1e-4, 2e-4, 5e-4
LoRA, attention + MLP	8, 16, 32, 64, 128	1e-4, 2e-4, 5e-4
full fine-tune	all 1,574.9 M parameters	5e-6, 1e-5, 2e-5

Table 15: The grid. Every cell is run on both mixes. 1e-3 is absent on purpose, see below.

CONFIG	MIX	WILD WORDS	WILD LOOP	RECOVERED	LS WER	FLEURS CER
base large-v3	none	0.999	0.008	0.698	0.035	0.305
LoRA r8 attn+mlp, 1e-4	no_synth	0.961	0.012	0.594	0.035	0.633
	all	0.745	0.013	0.716	0.036	0.541
LoRA r8 attn+mlp, 2e-4	no_synth	0.969	0.011	0.587	0.034	0.603
	all	0.758	0.013	0.762	0.035	0.507
LoRA r8 attn+mlp, 5e-4	no_synth	0.968	0.008	0.595	0.035	0.530

Table 16: The paired sweep, all 33 configurations. wild words is the share of 4,421 real voice-free clips where the model emitted words, wild loop the share carrying a token run of six or more, recovered the share of genuinely spoken phrases it got back. Lower is better everywhere except recovered.

Figure 13: Hallucination on real voice-free audio, with and without the synthetic positives, at every configuration in the grid.

Figure 14: The trade from Figure 1 , redrawn with the fine-tunes on it. The mix without the synthetic positives walks left and down. The mix with them walks left and up.

Figure 15: Training curves for all 66 runs. The mix without the synthetic positives sits below the mix with them for the whole of training, and it is the worse model on every axis that matters. An empty target is cheap to predict, and no_synth is 77% blank against all at 47%, so the gap is arithmetic rather than evidence. Full fine-tunes (dashed) are still descending at step 1,000.

Figure 16: Final training loss against phrase recovery. The two mixes barely overlap on the loss axis and the ranking inverts across them, so loss cannot be used to choose between them. Within a mix it carries little signal either.

Figure 17: Same rank, same learning rate, same mix, different modules. Adapting the two feed-forward matrices as well as the four attention projections is what makes the model stop. Attention-only adapters barely move off the base model at any rank.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 480 papers

Sep 3, 2026

Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection

Whisper is a widely used foundation model for automatic speech recognition (ASR), but its generative decoder can produce fluent hallucinated transcripts for inputs containing little or no speech. We propose a training-free, inference-time method to reduce these hallucinations using low-rank projection of decoder activations. A compact hallucination-associated subspace is estimated from non-speech calibration data, and decoder hidden states are projected away from this subspace during inference. We evaluate two variants: always-on, which applies projection to all inputs, and gated, which applies it only when Whisper predicts that an input is likely non-speech. Across non-speech benchmarks, always-on projection reduces average hallucination rate (HR) from 31.31% to 2.44%, a 92.21% relative reduction, while gated projection reduces HR to 3.74%, an 88.05% relative reduction, with lower false rejection of genuine speech. On LibriSpeech, gated projection increases absolute word error rate (WER) by 0.33-4.39 percentage points and yields false-rejection rates (FRR) of 0.41--9.97% across model and split settings. These results show that low-rank activation projection can substantially suppress Whisper hallucinations without retraining, while providing a controllable trade-off between hallucination suppression and speech recognition performance.

Maryam Abbasihafshejani, Murtuza Jadliwala

The university of Texas at San Antonio

Speech Foundation Models Automatic Speech Recognition Activation Steering

Ask PDF

Jun 5, 2026

Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders

Whisper, a widely adopted ASR model, is known to suffer from hallucinations - coherent transcriptions generated for non-speech audio entirely disconnected from the input. We investigate whether hallucinations can be detected and mitigated through Whisper's internal representations. We extract audio encoder activations and evaluate two representation spaces: raw Whisper activations and Sparse AutoEncoder (SAE) latents. We show that both spaces encode linearly separable hallucination-related information, with discriminative power concentrated in a sparse feature subset and increasing toward deeper encoder layers. We propose two steering strategies: activation-space steering and SAE latent-space steering. SAE-based steering reduces hallucination rate from 72.63% to 14.11% for Whisper small and from 86.88% to 27.33% for Whisper large-v3 on the full non-speech test set, with small WER degradation on speech data, approaching the performance of fine-tuning-based methods.

Georgii Aparin, Vadim Popov, Tasnima Sadekova +1

AI Foundation and Algorithm Lab · National Research University Higher School of Economics

Audio-Language Model Hallucination Detection Sparse Autoencoders Automatic Speech Recognition Representation Engineering Representation Steering

Ask PDF

Jun 22, 2026

From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection

Hallucinations of ASR models - fluent transcriptions with no basis in audio - degrade system performance and pose risks in downstream applications. Robust detection of such errors remains a challenge. This paper studies Whisper large v3 hallucination detection on real-speech human-annotated data across three paradigms: text-based, LLM-based, and internal decoder state probing. Text classifiers utilizing metrics for text evaluation achieve high recall but degrade without reference transcripts. LLM-based detection improves precision with domain-specific prompt conditioning, yet remains less competitive than the lightweight text-based methods. Probing Whisper's decoder representations, without a ground-truth reference, yields the strongest performance, revealing that hallucination traits are encoded across intermediate decoding layers. A late-fusion meta-classifier combining text and internal-state outputs achieves the best overall detection performance.

Jan Jasiński, Mateusz Barański, Julitta Bartolewska +2

Signal Processing Group, Institute of Electronics, AGH University of Krakow, Poland

Late Fusion Audio-Language Model Hallucination Detection Automatic Speech Recognition

Ask PDF

Abstract

Automatic Speech Recognition

Explore similar work

Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection

Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders

From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection