Whisper-Flash: Acoustically Conditioned Parallel Drafting for Faster Whisper Decoding

Huapeng Zhou; Huayu Wang; Junkai Wu; Kangqi Wang; Xinyu Wang

cs.SDSep 26, 2026

Whisper-Flash: Acoustically Conditioned Parallel Drafting for Faster Whisper Decoding

Authors: Huapeng Zhou, Huayu Wang, Junkai Wu, Kangqi Wang, Xinyu Wang

Organizations: Boson AI · University of Washington · Independent Researcher · McGill University

Abstract

Whisper is a widely used encoder-decoder model for speech recognition. Its encoder reads an utterance in one parallel pass, but its decoder writes the transcript one token at a time, which dominates inference time. Speculative decoding shortens such loops without changing their output: a small drafter guesses several upcoming tokens, and the original model verifies them all in one forward pass. We present Whisper-Flash, a two-layer drafter built on a property of speech recognition: the words still to be written have already been spoken. It reads Whisper's encoded audio and accepted decoder states and proposes eight tokens in a single forward pass. On the complete LibriSpeech test sets, Whisper-Flash processes $3.16\times/2.85\times$ as much audio per second as greedy decoding with identical outputs, and it remains faster at batch sizes up to 96 and under temperature sampling. Ablations show that direct access to the audio matters most.

Figures & tables

Panel of Figure 1 — Figure 1: Whisper-Flash drafts eight tokens in one pass from Whisper’s acoustic memory and accepted decoder states. The verifier accepts “a cat,” replaces “standing” with “sitting” and discards later guesses; the correction becomes the next anchor. Words stand in for token IDs.

64 development clips (427.2 s)
	Greedy decoding		Whisper-Flash
Batch	tok/s	audio s/s	tok/s	audio s/s	$S_{G}$
1	181.7	51.93	530.8	151.68	2.92
2	266.7	76.22	765.4	218.71	2.87
4	397.8	113.69	1052.0	300.64	2.64
8	554.8	158.54	1270.4	363.04	2.29

Table 1: Throughput by batch size on an H100, timed from waveform to text: generated text tokens (tok/s) and seconds of audio processed per second.

S_{G}

is the speedup over greedy decoding at the same batch size, whose tokens Whisper-Flash matches at every batch size.

Figure 2: Throughput versus batch size on the 64 development clips of Table 1 , timed from waveform to text.

Whisper large-v2 systems
Engine	Audio s/s $\uparrow$	WER (%) $\downarrow$
HF greedy	24.72	2.90
HF + length penalty	24.84	2.90
Whisper-Medusa Linear	35.82	5.80
Whisper-Medusa Block	32.14	3.79
Whisper-Medusa Block †	29.70	2.90

Table 2: Released Whisper-Medusa checkpoints and Whisper-Flash on the 64 clips of Table 1 (batch size one, waveform to text). Medusa uses large-v2, Hugging Face Transformers and its official acceptance rule;

\dagger

is our exact strict-argmax Medusa-Block.

Conditioning	$\alpha$ (%)	$\tau$	Audio s/s	$S_{G}$
Audio + history	47.37	4.60	157.07	2.81
Without direct audio	16.48	2.23	97.55	1.69
Without target history	30.15	3.29	128.63	2.30

Table 3: Conditioning ablation on 128 development clips, batch size one. The drafters share initialization, data, seed and budget (60.7M/47.5M/52.5M trainable parameters). Each drafter ran in its own job, and

S_{G}

is relative to greedy decoding timed in that job (55.9/57.7/55.9 audio s/s); all outputs match greedy decoding.

	Audio s/s			WER (%)
$T$	AR	WF	Gain	AR	WF
0	56.26	173.68	3.09	2.24	2.24
0.2	55.06	166.18	3.02	2.21	2.21
0.4	55.22	165.84	3.00	2.14	2.33
0.6	55.34	164.51	2.97	2.35	2.37
0.8	55.49	159.12	2.87	2.77	2.95

Table 4: Temperature sampling on 256 development clips (batch size one, three runs per clip). AR samples autoregressively from the target; gain is Whisper-Flash (WF) over AR.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

64 development clips
Batch	HF SDPA	FW core	Greedy	Whisper-Flash
1	19.26	38.33	51.93	151.68
2	26.66	55.72	76.22	218.71
4	36.15	82.02	113.69	300.64
8	47.13	104.71	158.54	363.04
16	55.46	121.77	194.42	403.88

Appendix

Table A1: Audio s/s of all engines by batch size (H100, waveform to text), the data behind Fig. 2 and Table 1 . FW denotes faster-whisper.

Batch	Request or group	Greedy (%)	Whisper-Flash (%)
1	p25 length	12.01	34.19
1	p50 length	8.51	20.94
1	p90 length	4.94	15.30
32	group 0	29.25	50.01
32	group 5	29.60	55.82
96	group 0	30.23	52.04

Appendix

Table A2: Encoder share of request time, from CUDA events that include host submission gaps. Batch one uses representative requests at the 25th, 50th and 90th length percentiles; larger batches use complete groups.

Duration (s)	Utterances	Speedup
$[0,5)$	2508	2.278
$[5,10)$	1948	3.045
$[10,20)$	965	3.807
$[20,30)$	122	4.024
$[30,\infty)$	16	3.280

Appendix

Table A3: Speedup over greedy decoding on the complete test sets by utterance duration (test-clean and test-other combined, batch size one). Recordings longer than 30 s are decoded as independent 30 s segments.

1,914 clips, 6,000 updates
Layers	Position	$\alpha$ (%)	$a$	ms
2	Abs.	9.68	0.77	97.4
2	RoPE	31.94	2.56	59.3
train-clean-100, 40,000 updates
2	Abs.	48.12	3.85	47.1
5	Abs.	52.72	4.22	47.8

Appendix

Table A4: Position encoding and depth, trained with matched data, seed and budget; 32 dev-clean clips, batch size one.

a

is the mean number of accepted candidates per round.

$K$	ms	$S_{G}$	$\alpha$ (%)	$\tau$
1	84.74	1.41	92.54	1.89
2	66.37	1.80	87.14	2.67
4	49.94	2.39	74.74	3.85
8	43.77	2.72	54.76	5.17

Appendix

Table A5: Verifying only the first

K

of the eight proposals of the same drafter; 64 development clips, batch size one, timed from waveform to text.

Continuation	First (%)	$\alpha$ (%)	$\tau$	ms
No continuation	82.02	46.45	4.52	41.53
train-clean-100 only	81.42	46.75	4.54	41.52
LibriSpeech (925.6 h)	87.69	53.95	5.09	38.84
+ TED-LIUM, VoxPopuli	87.24	54.21	5.12	38.77

Appendix

Table A6: Continuing the train-clean-100 drafter for 80,000 updates on different pools, evaluated on 128 dev-other clips at batch size one. “First” is first-candidate acceptance; the last row is the released drafter.

$T$	$\alpha$ (%)	$\tau$	Gain CI	$\Delta$ WER [CI]
0	57.53	5.39	[2.99, 3.19]	0.00 [0.00, 0.00]
0.2	57.20	5.37	[2.93, 3.11]	0.00 [-0.10, +0.09]
0.4	56.48	5.31	[2.91, 3.10]	+0.18 [+0.03, +0.33]
0.6	55.46	5.23	[2.88, 3.06]	+0.01 [-0.15, +0.17]
0.8	51.66	4.94	[2.78, 2.96]	+0.18 [-0.20, +0.59]
1	44.79	4.42	[2.58, 2.77]	-0.28 [-1.25, +0.61]

Appendix

Table A7: Temperature-sampling diagnostics for Table 4 . Gain CI is the 95% interval of the throughput gain over autoregressive sampling;

\Delta

WER is Whisper-Flash minus autoregressive sampling in percentage points, with paired 95% intervals.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 459 papers

Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding
Mar 6, 2026Hoseong Ahn, Jeongyun Chae, Yoonji Park +1Language Model Decoding Automatic Speech Recognition
Sungkyunkwan University, Republic of Korea
LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
Sep 29, 2026Hao-Yuan He, Peng-Fei Liu, Si Shen +1Long-Context Language Model Inference Blockwise Diffusion Language Models
Alignment Drift in Single-Model Speculative Decoding for ASR: Mechanism, Correction, and Cost
Aug 13, 2026Xinyu Wang, Huapeng Zhou, Ziyu Zhao +5Audio-Text Alignment Speculative Decoding
1Boson AI · 2McGill University

Mar 6, 2026J/K move · Enter open · S save

Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding

Hoseong Ahn, Jeongyun Chae, Yoonji Park, Kyuhong Shim

Sungkyunkwan University, Republic of Korea

Language Model Decoding Automatic Speech Recognition Contrastive Decoding Long-Form Speech Recognition

Open paper AskPDF arXiv

Long-form speech recognition with large encoder-decoder models such as Whisper often exhibit hallucinations, repetition loops, and content omissions. These errors can accumulate and be further amplified when the previous segment's transcription is used as decoding context. We propose Whisper-CD, a training-free contrastive decoding framework that contrasts clean-audio logits against negative logits computed from three acoustically motivated perturbations: Gaussian noise injection, silence signal, and audio temporal shift. We aggregate these negatives via the log-sum-exp operator, building a unified multi-negative objective for token-by-token decoding. Across five English long-form benchmarks, Whisper-CD reduces WER by up to 24.3pp on CORAAL and shows 48% faster token generation throughput than beam search. Because Whisper-CD operates purely at inference time, it can be applied as a drop-in replacement to already-deployed Whisper systems without retraining.

Abstract

Automatic Speech Recognition

Explore similar work

Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding

LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter

Alignment Drift in Single-Model Speculative Decoding for ASR: Mechanism, Correction, and Cost