The Null Token Knows: Reducing Message-Free Hallucination in ASR and NMT

Kirill Borodin; Vasiliy Kudryavtsev; Ivan Viakhirev; Grach Mkrtchian

cs.CLAug 16, 2026

The Null Token Knows: Reducing Message-Free Hallucination in ASR and NMT

Authors: Kirill Borodin, Vasiliy Kudryavtsev, Ivan Viakhirev, Grach Mkrtchian

Organizations: Lab260, Moscow, Russia · BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia · ITMO, Saint Petersburg, Russia ·

Abstract

Modern encoder-decoder systems can produce fluent text even when their input contains no recoverable message. We study this failure in ASR and NMT through the models' reserved null tokens, asking whether the score for ending generation already carries a usable abstention signal. Across speech recognizers and translation models, we audit native null-token scores and scalar logit shifts. In Whisper, we additionally probe decoder states and compare supervised row edits with conventional external gates. The evaluated models often expose a useful abstention signal, but stock decoding does not reliably act on it. Raising the null-token score can sharply suppress fabrication, but aggressive intervention also deletes valid speech or shortens legitimate translations. These findings turn the null token into a diagnostic lens on hallucination and motivate evaluating abstention methods by both suppression and deletion costs, rather than by hallucination reduction alone.

Figures & tables

Figure 1: Developmental clip-stratified probes separate non-speech from speech after the first Whisper decoder block. The layer-0 embedding is at chance; the gray band is the permutation baseline. Folds are not grouped by source corpus, so this does not establish source-independent encoding.

Figure 2: Developmental native null-token readout at decode step 0. (a) Whisper abstain-margin AUROC across the 1060-clip battery and decoder depth. (b) Class-conditional mean final margins on each model’s named developmental subsets; connecting lines join the two means and zero is the abstention boundary. Subset denominators are in the supplement.

Model	probe	ns	EOT	$\cos(\Delta,w_{\rm ns})$	random $p_{95}$	ns $\to$ EOT
	state	AUROC@0		row/ns geometry		graft
tiny	1.00	0.953	0.977	$+0.119$	0.103	fails
base	1.00	0.963	0.840	$+0.043$	0.092	fails
small	1.00	0.878	0.835	$+0.114$	0.112	fails
medium	1.00	0.915	0.728	$+0.034$	0.026	fails
turbo	1.00	0.988	0.995	$-0.014$	0.046	fails

Table 1: Developmental Whisper state-probe and native-logit AUROC at step 0. “ns” is Whisper’s no-speech token and EOT is end-of-transcript.

\Delta

is the trained EOT-row update;

\cos(\Delta,w_{\rm ns})

compares it with the native no-speech row, and random

p_{95}

is the 95th percentile of matched random-direction cosines. The final column tests grafting the no-speech row direction into EOT. Cross-architecture readouts are in supplementary Table 10 .

Developmental method	LOR	speech rejected/empty
native EOT margin	89.8	$\leq 10.0$
native no_speech_prob	4.3	$\leq 10.0$
frozen-state logistic gate	0.0	$\leq 10.0$
FireRedVAD front-end	1.3	$\leq 10.0$
stock checkpoint	96.2	0.0
EOT bias $b=5$	24.3	7.4

Table 2: Matched but inspected Whisper-small comparison on 630 non-speech and 430 speech clips (percent). For the four thresholded detectors, the review bundle preserves only that the selected point satisfies the

\leq 10\%

cap, not the exact speech-side rate. The cap is illustrative and rows are descriptive points, not efficacy or superiority estimates.

Method	$F$	$D$	$C_{0.5}$	$C_{1}$	$C_{2}$	$C_{5}$	$C_{10}$
stock	32.5	1.6	33.3	34.1	35.7	40.4	48.3
trained EOT row	4.3	33.0	20.8	37.3	70.4	169.5	334.8
EOT bias $b=5$	10.1	17.6	18.9	27.7	45.3	98.0	185.7
EOT bias $b=6$	6.4	34.6	23.7	41.0	75.8	179.9	353.6
analytic graft $\lambda=2$	35.4	17.4	44.1	52.8	70.1	122.1	208.8

Table 3: Locked conditional-rate sensitivity

C_{\kappa}=F+\kappa D

(lower is better).

F

is total words retained by the specified lexical normalizer divided by total non-speech audio minutes;

D

is total reference-word deletions divided by total speech audio minutes. Component values are rounded to one decimal; displayed costs come from unrounded values. Methods and weights were author-frozen before this 300/300 decode. Boldface marks the descriptive minimum on this sample, not statistical superiority or deployment utility; rank uncertainty is unquantified.

Method	Locus	Params	0% HR at	WER
eot_row	EOT row	$\mathbf{+768}$	50	5.11
LoRA ( $r{=}8$ )	q/v proj.	884,736	50	5.5
decoder-only FT	decoder	154M	50	5.6
full FT	all	242M	50	5.7
random-matched	random scal.	95,232	300	5.2
LayerNorm (all)	LN	95,232	150	7.6

Table 4: Cost comparison on Whisper small. Training rows report the first zero-HR step and held-out probe WER (stock

5.11

);

50^{+}

also requires per-model head screening. a The released AudioSAE is evaluated directly on our

630

-clip battery and test-clean protocol (stock WER

3.73

\alpha=1

leaves HR at

94.6\%

, whereas the first zero-HR point,

\alpha=8

, gives WER

100\%

(full sweep in supplementary Table 26 ). b The reduced LLT layer-mixing proxy raises HR

92\to 100\%

and is not a full teacher-distillation reproduction.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Result	Fit/calibration pool	Selection rule or pool	Reporting pool	Inspected?
Frozen-state probe	Full battery within five clip folds	Cross-validated probe; PCA fitted within each analysis	The same 630 non-speech and 430 speech clips	yes
Trained EOT row	MUSAN calibration half, synthetic silence, LibriSpeech dev-clean	Common step 50; first-zero rows summarize trajectories	Full 630-clip non-speech battery and named speech subsets	yes
Analytic row graft	Disjoint MUSAN half and LibriSpeech dev-clean speakers	Admissible interval on the calibration pool	Stress battery and test-clean, disjoint from fitting	yes
AED/NMT scalar bias	No parameter fit; margins logged on fixed dose-response data	Bias chosen on 150 non-speech and 60 speech clips	The same dose-response subset; MT quality on WMT19	yes
Common-cost comparison	Each method’s stated calibration	Lowest LOR satisfying the illustrative $10\%$ speech-side cap	The same 630 non-speech and 430 speech clips	yes
Multilingual row/scalars	Seven Whisper tags or named language calibration splits	Frozen row/scalars after calibration	Eight held-out tags or held-out language clips	yes

Appendix

Table 5: Fit, selection, and reporting roles. “Inspected” means that reporting-pool outputs were available during method development. The locked cost ledger entry is the only post-freeze reporting pool; exact source IDs and computed collisions are author-reported as stored in project manifests that are not linked from the review PDF.

Method	LOR	speech-side rate	within cap
native EOT margin	89.8	$\leq 10.0$	yes
native no_speech_prob (SOT)	4.3	$\leq 10.0$	yes
logistic gate (frozen state)	0.0	$\leq 10.0$	yes
FireRedVAD front-end	1.3	$\leq 10.0$	yes
stock	96.2	0.0	yes
constant EOT bias $b=5$	24.3	7.4	yes

Appendix

Table 6: Matched, inspected Whisper-small comparison (percent). LOR is the normalized lexical-output rate. Speech-side rate is rejection for detectors and empty output for checkpoint interventions. The external detectors have lower developmental LOR while satisfying the coarse cap; this is not a locked efficacy or superiority comparison.

Model	stock LOR	patched LOR	patched WER (stock)	$\lambda^{\star}$	$[\lambda_{\mathrm{lo}},\lambda_{\mathrm{hi}}]$	in?	LOR	test-clean WER
	trained EOT row			analytic graft
tiny	94.9	0.0	10.61 (10.61)	1.017	$[0.45,2.28]$	yes	0.3	$7.54\to 8.22$
base	99.7	0.0	7.74 (7.74)	0.507	$[0.51,\infty)$	yes	2.5	$5.04\to 4.84$
small	96.2	0.0	5.11 (5.11)	2.406	$[0.53,10.91]$	yes	0.5	$3.44\to 4.16$
medium	86.8	0.0	3.59 (3.59)	0.497	$[0.50,\infty)$	yes	1.3	$2.88\to 7.60$
turbo	74.3	0.0	3.59 (2.69)	0.661	$[0.66,\infty)$	yes	3.5	$2.16\to 2.26$

Appendix

Table 7: Developmental trained-row and analytic-graft summary by Whisper scale. Trained-row LOR is on the inspected battery at common step 50; WER is on the held-out dev-clean probe and parentheses give stock WER. The analytic graft uses disjoint calibration pools and reports battery LOR plus test-clean WER. Values and interval bounds are rounded independently.

Method	$F$	$D$	$C_{0.5}$	$C_{1}$	$C_{2}$	$C_{5}$	$C_{10}$
stock	32.5	1.6	33.3	34.1	35.7	40.4	48.3
trained EOT row	4.3	33.0	20.8	37.3	70.4	169.5	334.8
constant EOT bias $b=5$	10.1	17.6	18.9	27.7	45.3	98.0	185.7
constant EOT bias $b=6$	6.4	34.6	23.7	41.0	75.8	179.9	353.6
analytic graft $\lambda=2$	35.4	17.4	44.1	52.8	70.1	122.1	208.8

Appendix

Table 8: Locked

C_{\kappa}

(lower is better).

F

and

D

are rounded to one decimal; costs use unrounded components. Boldface marks the descriptive minimum on this 600-clip sample. Rank uncertainty is unquantified, the weights are illustrative, and external gates were not run on this set.

Model	stock LOR	fixed LOR	worst-language LOR	median empty	median $\Delta$ CER
large-v3	81.8	8.3	pt: 20.0	0.0	1.7
turbo	77.6	5.8	es: 11.7	0.0	0.1
base	92.3	3.1	ko: 46.7	29.1	24.7
small	79.0	2.0	ko: 8.3	47.5	46.4
tiny	71.8	3.8	ko: 56.7	72.5	56.7
medium	75.0	0.3	ko: 3.3	84.2	84.2

Appendix

Table 9: Held-out 15-language Whisper evaluation (percent). Large-v3 and turbo retain real speech at the calibrated point; smaller scales reach low mean LOR by abstaining heavily. Korean is the worst residual language at all six scales.

Model	Task	AUROC 0	lexical LOR	Speech/MT change	$\beta$	$\beta^{\star}$ pred.	$\beta^{\star}$ emp.
Canary-1B	ASR	$1.000$	$\mathbf{79.3\to 0.0}$	WER $\mathbf{7.02\to 6.94}$	$8$	8.0	8.0
OWSM v3.1	ASR	$0.998$	$90.7\to 1.3$	WER $8.19\to 20.9$	$10$	10.0	9.0
NLLB-200	NMT	$0.999$	$27.5\to 1.0$	COMET $83.4\to 80.9$	$6$	3.5	6.0
MarianMT	NMT	$0.911$	$7.0\to 0.0$	COMET $79.9\to 75.6$	$6$	9.5	3.0

Appendix

Table 10: Developmental scalar null-token-bias summary (percent except

\beta

). AED dose response uses 150 non-speech and 60 speech clips; NMT quality uses real-source WMT19.

\beta^{\star}

is selected and reported on the same grid, so these are descriptive rather than locked estimates. COMET is multiplied by 100.

Figure 3: Developmental Canary-1B dose response on 150 non-speech and 60 speech clips. Curves are predicted from step-0 margins; points are measured decodes. The dashed line marks the descriptive error-balance grid point, where missed and false abstentions are closest.

Model	slope	$95\%$ CI	$r$
tiny	0.96	$[0.41,1.50]$	0.90
base	0.80	$[0.52,1.08]$	0.96
small	1.05	$[0.65,1.45]$	0.95
medium	1.53	$[1.01,2.06]$	0.96
turbo	1.89	$[1.25,2.52]$	0.96
large-v3	1.69	$[1.18,2.21]$	0.97

Appendix

Table 11: Developmental replicated analysis of learned abstention margin versus

\mathrm{logit}(p)

over seven training fractions per scale. These run-level intervals summarize the evaluated trainings; heterogeneity is high and equivalence to unit slope is not established.

Category	Source	$n$
silence_pure	synthetic zero $+$ dither	40
silence_room	pink noise (room tone)	40
noise_stationary	MUSAN noise	150
noise_events	ESC-50	150
noise_urban	UrbanSound8K	150
music	MUSAN music (instrumental)	100

Appendix

Table 12: The 1060-clip stress battery (3.1 h, 16 kHz mono, clips

\leq 30

s; seed 20260611). Non-speech strata carry an empty reference; speech strata are FLEURS-derived. Exclusions: UrbanSound8K classes that may contain words (children_playing, street_music) and vocal MUSAN tracks are dropped. Speech sources are RMS-normalized to

-20

dBFS.

		test-clean	Lexical-output rate (%) by stratum
Model	Type	WER %	silence	room	noise	events	urban	music	all
Whisper tiny	AED	7.54	100	100	86	96	99	97	94.9
Whisper base	AED	5.04	100	100	100	99	100	100	99.7
Whisper small	AED	3.44	100	50	97	100	100	100	96.2
Whisper medium	AED	2.88	0	0	99	100	100	98	86.8
Whisper turbo	AED	2.16	25	42	82	67	83	94	74.3

Appendix

Table 13: Stock behavior of the fifteen-model core zoo. LibriSpeech test-clean WER (Canary-1B and OWSM v3.1 on

500

utterances, others on

2620

) validates the pipeline. Lexical-output rate is the share of non-speech clips with at least one retained letter after normalization. Exact repository revisions are author-reported as listed in a prepared model manifest that is not linked from the review PDF.

Use	Evaluation set	Reason
zoo/stats	full $630$ NS $+$ $430$ speech	headline rates/CIs
deployable ROC	fixed $300$ NS $+$ $430$ speech	threshold sweeps
AED/NMT bias	fixed $150$ NS $+$ $60$ speech	dose response
degraded ledger	fixed ID degradations	error alignment
spontaneous cost	Earnings-22/MUSAN-speech	external domain
long form	$40$ clips, $74$ min	chunk boundaries

Appendix

Table 14: Evaluation-subset map. Subsets differ because large-model threshold sweeps are substantially more expensive than one-pass headline evaluation. Every intervention is compared with stock re-decoded on the same subset; rates are not compared across rows. The first three subsets are fixed and nested only where their listed counts coincide. NS denotes non-speech.

Model	LOR	$95\%$ CI	LOR	$95\%$ CI	LOR	$95\%$ CI
	stock		eot_row @50		analytic
tiny	94.9	$[92.9,96.4]$	0.0	$[0.0,0.6]$	0.3	$[0.1,1.2]$
base	99.7	$[98.8,99.9]$	0.0	$[0.0,0.6]$	2.5	$[1.6,4.1]$
small	96.2	$[94.4,97.4]$	0.0	$[0.0,0.6]$	0.5	$[0.2,1.4]$
medium	86.8	$[84.0,89.2]$	0.0	$[0.0,0.6]$	1.3	$[0.6,2.5]$
turbo	74.3	$[70.7,77.5]$	0.0	$[0.0,0.6]$	3.5	$[2.3,5.2]$

Appendix

Table 15: Normalized lexical-output rates (LOR, %;

n{=}630

non-speech clips) with clip-level Wilson

95\%

intervals ( Wilson 1927 ) . These conditional finite-battery summaries are not source-clustered or corpus-transfer intervals. The trained row is evaluated at common step 50; paired tests are in Table 16 .

Model	$b$	exact $p$	Holm $p$	OR
tiny	598	$2{\times}10^{-180}$	$8{\times}10^{-180}$	1197
base	628	$2{\times}10^{-189}$	$1{\times}10^{-188}$	1257
small	606	$8{\times}10^{-183}$	$4{\times}10^{-182}$	1213
medium	547	$4{\times}10^{-165}$	$1{\times}10^{-164}$	1095
turbo	468	$3{\times}10^{-141}$	$3{\times}10^{-141}$	937
large-v3	485	$2{\times}10^{-146}$	$4{\times}10^{-146}$	971

Appendix

Table 16: Exact paired McNemar tests ( McNemar 1947 ) for stock versus eot_row @50 on all

630

non-speech clips.

b

is stock-nonempty/patch-empty and

c

the reverse. Holm correction is over the six-scale family; OR uses the Haldane correction because every

c=0

. All learned-patch LORs are

0.0\%

(Wilson CI

[0.0,0.6]

). The independently decoded analytic patch likewise has

c=0

on every scale and exact

p<10^{-130}

; its per-scale LORs are in Table 15 . These clip-level tests are conditional on the inspected battery and do not address method-selection or source-transfer uncertainty.

Model	grid runs	max residual LOR	probe-WER range
tiny	9	0.0	$11.1$ – $11.9$
base	9	0.0	$8.3$ – $9.3$
small	9	2.0	$5.1$ – $9.0$
medium	9	0.0	$3.9$ – $63.1$
turbo	9	0.0	$2.4$ – $11.4$
large-v3	9	0.0	$2.2$ – $3.7$

Appendix

Table 17: Matched-capacity random-subnetwork verification on all six scales: three learning rates by three mask seeds per scale. Max residual LOR is the worst final normalized lexical-output rate across the nine runs. Reachability is scale-wide (at most

2\%

), but WER safety is not: medium is acutely learning-rate-sensitive. This supports no privileged parameter locus, not interchangeable operating points.

$p$ (non-speech)	$\mathrm{logit}(p)$	abstain margin	$\\|\Delta\\|$ row
0.20	$-1.39$	12.67	0.359
0.35	$-0.62$	14.98	0.379
0.50	$+0.00$	14.46	0.373
0.65	$+0.62$	14.63	0.366
0.80	$+1.39$	16.10	0.353

Appendix

Table 18: Raw base-rate sweep behind Table 11 (Whisper small): converged step-0 abstain margin and unembedding row-delta norm at each training non-speech fraction

p

. Fitted slope

0.98

\mathrm{logit}(p)

(

r=0.85

;

95\%

[-0.12,2.09]

n{=}5

); the row norm stays nearly constant; the row controls the gain while the base rate changes the threshold.

	Stock margin			Patched margin
Model	0	10	30	0	10	30
tiny	$-6.4$	$-1.3$	$-3.4$	$+14.5$	$+6.5$	$+4.9$
base	$-6.2$	$-1.0$	$-2.6$	$+15.2$	$+6.1$	$+5.9$
small	$-7.8$	$-3.1$	$-2.5$	$+16.2$	$+4.1$	$+5.8$
medium	$-8.0$	$-3.3$	$-2.5$	$+18.9$	$+5.4$	$+10.2$
turbo	$-4.6$	$-3.4$	$-4.9$	$+12.0$	$+2.2$	$-0.8$

Appendix

Table 19: Position-wise abstain margin

\mathrm{logit}(\mathrm{EOT})-\max_{t}\mathrm{logit}(\mathrm{content}_{t})

at emitted positions

0

10

30

, from teacher-forcing each stock model on its own hallucinated transcript. Stock margins are negative almost everywhere; the patch lifts them most at early positions, and residual late negativity on turbo and large-v3 explains why the position-0 intervention does not transfer directly to long-form decoding.

Model	$\Delta$ Sub	$\Delta$ Del	$\Delta$ Ins	$\Delta$ WER
tiny	$-16.1$	$+24.3$	$-17.7$	$-9.5$
base	$-7.8$	$+12.4$	$-13.7$	$-9.1$
small	$-3.8$	$+10.0$	$-0.8$	$+5.5$
medium	$-3.8$	$+13.4$	$-1.3$	$+8.4$
turbo	$-2.9$	$+12.4$	$-0.7$	$+8.8$
large-v3	$-3.0$	$+15.1$	$-0.6$	$+11.6$

Appendix

Table 20: Fixed-subset decomposition of the degraded-speech WER change under the trained EOT row, in percentage points. Tiny and base remove enough substitutions and insertions to offset added deletions; the four stronger scales over-delete.

Variant	gap words/min	deletions	WER
stock	62.0	2.5	7.4
eot_row @50	27.6	14.5	21.1
VAD mute $+$ stock	81.2	2.2	16.0
VAD segmentation $+$ stock	4.7	0.3	4.5
boundary sweep	$26.3$ – $72.9$	$1.1$ – $15.6$	$22.0$ – $54.6$

Appendix

Table 21: Whisper turbo on 40 clips (38.7 gap-minutes). The earlier

81.2

row muted gaps but decoded fixed windows; it was mislabeled as segmentation. Correct VAD segmentation cuts and independently decodes speech regions, dominating the null patch on both frontier axes. Rates are percent except gap words/min.

	LayerNorm @ best-zero step			eot_row @ 50
Model	WER	deletions	empty $/100$	WER	deletions	empty $/100$
tiny	$23.7\to 50.9$	$5.7\to 41.8$	20	$23.7\to 63.1$	$5.7\to 55.6$	22
base	$16.5\to 21.4$	$4.2\to 9.9$	7	$16.5\to 20.6$	$4.2\to 9.6$	2
small	$13.2\to 23.4$	$5.1\to 15.3$	10	$13.2\to 23.1$	$5.1\to 16.6$	5
medium	$11.7\to 12.1$	$3.8\to 4.2$	3	$11.7\to 14.9$	$3.8\to 8.0$	0
turbo	$13.8\to 25.0$	$4.0\to 15.0$	30	$13.8\to 13.4$	$4.0\to 4.3$	27

Appendix

Table 22: Spontaneous-speech cost and checkpoint-selection risk. Earnings-22 ( Del Rio et al. 2022 ) WER and deletion rates (stock

\to

patched, %) and empty outputs per 100 spontaneous MUSAN-speech clips, for the LayerNorm regime at its best zero-hallucination checkpoint and eot_row at step 50. Clean-probe checkpoint selection is blind to this: late checkpoints leave up to

62\%

of spontaneous speech empty (large-v3, LayerNorm at step 350); tiny pays

+27

–

+39

pp WER under any zeroing checkpoint; medium (LayerNorm) and turbo ( eot_row ) are nearly free.

Lang	split	stock	patched	WER
en	pilot	63	1.3	6.3
ru	pilot	100	0.8	n.a.
de	pilot	81	1.0	13.9
ja	pilot	100	2.0	n.a.
hi	pilot	86	0.2	n.a.
zh	pilot	100	0.7	n.a.

Appendix

Table 23: Whisper-small multilingual EOT row: hallucination % on seven calibration tags and eight held-out tags, plus real-speech WER where word-based scoring was available. Seven held-out tags remain at

1.5

–

3.0\%

; Korean is the visible

13.3\%

exception. This is multilingual calibration followed by language holdout, not English zero-shot transfer.

Lang	Script	langid (stock $\to$ fix)	WER (stock $\to$ fix)
en	Latin	$0.38\to\textbf{1.00}$	$0.048\to 0.048$
de	Latin	$0.28\to\textbf{0.96}$	$0.116\to 0.116$
fr	Latin	$0.59\to\textbf{0.97}$	$0.170\to 0.170$
es	Latin	$0.19\to\textbf{0.91}$	$0.081\to 0.081$
it	Latin	$0.06\to\textbf{0.80}$	$0.124\to 0.124$
nl	Latin	$0.56\to\textbf{0.95}$	$0.188\to 0.188$

Appendix

Table 24: One row per language token restores Whisper small’s short-clip (

2

s) language identification without touching transcription: langid accuracy and word WER (stock

\to

fix) across fifteen FLEURS languages and seven scripts. Mean langid rises

0.29\to 0.94

; WER is bit-identical (the edit is scoped to the detection step). † ja/zh word-WER is

\approx 1

by construction (no word boundaries); only its invariance under the fix matters.

Model	Intervention	Halluc. (%)	WER
wav2vec2 (CTC)	anchor FT	$14.4\to\mathbf{0.8}$	$2.30\to 2.30$
HuBERT (CTC)	anchor FT	$61.4\to\mathbf{0.5}$	$2.24\to 4.66$
WavLM (CTC)	bias $\beta{=}2$	$36.2\to\mathbf{14.0}$	$9.15\to 10.77$
wav2vec2-base (CTC)	bias $\beta{=}2$	$30.8\to\mathbf{16.8}$	$4.15\to 5.72$
Conformer (CTC)	bias $\beta{=}2$	$11.1\to\mathbf{4.3}$	$2.75\to 3.82$
Emformer (RNN-T)	bias $\beta{=}4$	$24.8\to\mathbf{7.3}$	$7.13\to 8.14$

Appendix

Table 25: The blank is the CTC/RNN-T null token. Hallucination and WER are stock

\to

patched (%); “bias” is a single blank-logit scalar. Strong CTC systems reach near-zero via anchor fine-tuning; weaker systems reduce monotonically under one scalar.

Released AudioSAE-small condition	HR	$95\%$ CI	WER	$\Delta$ WER
stock (SAE disabled)	96.2	$[94.4,97.4]$	3.73	$+0.00$
reconstruction ( $\alpha=0$ )	96.2	$[94.4,97.4]$	3.41	$-0.32$
steering $\alpha=1$	94.6	$[92.6,96.1]$	3.74	$+0.01$
steering $\alpha=2$	95.1	$[93.1,96.5]$	61.33	$+57.60$
steering $\alpha=4$	48.9	$[45.0,52.8]$	188.57	$+184.85$
steering $\alpha=8$	0.0	$[0.0,0.6]$	100.00	$+96.27$

Appendix

Table 26: Common-axis evaluation of the released AudioSAE-Whisper-small final-layer encoder SAE ( Aparin et al. 2026b ) . We select the top 128 features by a logistic regression fit only on training pools and apply the released steering recipe. At the paper-strength setting

\alpha=1

, HR barely moves; increasing strength destroys transcription before suppressing fabrication. Zero HR is reached only when WER is

100\%

. One SAE layer replicated across the twelve encoder layers contributes

113.3

M parameters, versus

768

for the EOT row.

Model	Method	Screening	Params	HR	WER
Canary	stock	–	–	77.5	1.66
Canary	Calm top-8 head ablation	192 passes	0	72.9	1.70
Canary	EOS scalar, $\beta=8$	none	1	0.0	6.94
OWSM	stock	–	–	100.0	16.11
OWSM	Calm top-8 head ablation	288 passes	0	100.0	16.08
OWSM	EOS scalar, $\beta=9$	none	1	4.7	12.79

Appendix

Table 27: Cross-architecture Calm-Whisper route versus a one-scalar EOS edit. Screening all candidate decoder heads finds a maximum step-0

\Delta P(\mathrm{EOS})

of only

0.0006

on Canary and

0.0001

on OWSM; ablating the selected heads consequently has little or no effect. Canary uses

630

non-speech clips and

500

test-clean utterances. The slower OWSM ablation uses

300/150

with maxlenratio=0.4 , so its absolute WER is not comparable with other OWSM tables, but its stock/ablation comparison is paired. Scalar rows report their independently measured operating points.

OWSM EOS bias $\beta$	HR	WER
0.0	90.7	8.19
2.0	84.0	8.19
4.0	72.7	8.19
6.0	58.0	8.92
8.0	16.7	9.43
8.5	9.3	11.04

Appendix

Table 28: Fine OWSM dose response on the fixed

150

non-speech plus

60

speech subset. The bias is a monotone hallucination control, but there is no free operating point: the empirical balance point is

\beta=9

, whereas the

1.3\%

HR headline at

\beta=10

more than doubles WER.

Model	$\beta$	BLEU	chrF	COMET	nonempty	length
NLLB en $\to$ ru	$0$	29.0	54.4	83.4	100	91
NLLB en $\to$ ru	$6$	25.2	50.8	80.9	100	82
NLLB en $\to$ ru	$8$	13.9	37.4	70.3	100	54
Marian en $\to$ de	$0$	37.7	63.7	79.9	100	97
Marian en $\to$ de	$6$	34.5	59.0	75.6	100	87
Marian en $\to$ de	$10$	0.0	5.9	32.2	87	8

Appendix

Table 29: Real-source WMT19 translation quality under EOS bias. BLEU ( Papineni et al. 2002 ) , chrF ( Popović 2015 ) , and COMET ( Rei et al. 2020 ) ( wmt22-comet-da , multiplied by

100

) decline at the working point (

\beta=6

) and decline sharply under stronger early-termination bias. Nonempty and length are percentages.

Figure 4: Cross-architecture logit-lens depth traces at decode step 0. Curves show the mean EOS abstain margin for message-free audio or degenerate source text, clean input, and degraded input; bands are normal-approximation 95% confidence intervals over $n=120$ clips per condition for Canary/OWSM and $n=200$ sources per condition for NLLB/Marian. The dashed line is the EOS argmax boundary. Message-free input finishes above clean input in every model, but below zero. Marian uniquely promotes EOS in the middle stack and demotes it in the final block, delimiting the stronger claim that abstention is never internally promoted.

Hallucination in Language Models

Hallucination in language models studies how systems produce unsupported or false claims and how to detect, explain, and reduce them.

See all 247 papers

From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection
Jun 22, 2026Jan Jasiński, Mateusz Barański, Julitta Bartolewska +2Late Fusion Audio-Language Model Hallucination Detection
Signal Processing Group, Institute of Electronics, AGH University of Krakow, Poland
The Anatomy of an ASR Hallucination
Sep 3, 2026Hamees Sayed, Apoorv Singh, Kumar Aman +1Robust ASR Conformer
Smallest AI · Fast Code AI
Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders
Jun 5, 2026Georgii Aparin, Vadim Popov, Tasnima Sadekova +1Audio-Language Model Hallucination Detection Sparse Autoencoders
AI Foundation and Algorithm Lab · National Research University Higher School of Economics

Jun 22, 2026J/K move · Enter open · S save

From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection

Jan Jasiński, Mateusz Barański, Julitta Bartolewska, Marcin Witkowski+1

Signal Processing Group, Institute of Electronics, AGH University of Krakow, Poland

Late Fusion Audio-Language Model Hallucination Detection Automatic Speech Recognition

Open paper AskPDF arXiv

Hallucinations of ASR models - fluent transcriptions with no basis in audio - degrade system performance and pose risks in downstream applications. Robust detection of such errors remains a challenge. This paper studies Whisper large v3 hallucination detection on real-speech human-annotated data across three paradigms: text-based, LLM-based, and internal decoder state probing. Text classifiers utilizing metrics for text evaluation achieve high recall but degrade without reference transcripts. LLM-based detection improves precision with domain-specific prompt conditioning, yet remains less competitive than the lightweight text-based methods. Probing Whisper's decoder representations, without a ground-truth reference, yields the strongest performance, revealing that hallucination traits are encoded across intermediate decoding layers. A late-fusion meta-classifier combining text and internal-state outputs achieves the best overall detection performance.

Abstract

Hallucination in Language Models

Explore similar work

From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection

The Anatomy of an ASR Hallucination

Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders