WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

Yiming Yao; Chenyang Lyu; Xuanfan Ni; Longyue Wang; Weihua Luo; Yazheng Yang; Jinsong Su

cs.CLAug 24, 2026

WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

Authors: Yiming Yao, Chenyang Lyu, Xuanfan Ni, Longyue Wang, Weihua Luo, Yazheng Yang, Jinsong Su

Organizations: School of Informatics, Xiamen University · Alibaba Group

Abstract

Long-form audio inputs make the KV cache the dominant memory cost of speech LLMs. Prefill-only KV compression methods permanently discard audio KV positions once evicted, with no pathway to recover them during decoding. We show this is fragile on long-form audio: prefill attention concentrates near the audio start (an attention-sink effect), while decode-time attention distributes broadly, and the two rankings overlap weakly. We propose WnW (Waxing-and-Waning KV cache), which classifies KV-heads into anchor, tidal, and fixed roles via offline calibration. Anchor heads keep all audio KV on GPU and yield a decode-time signal of which audio region each token is read from; tidal heads keep a CPU-resident complement that is recalled chunk-by-chunk based on aggregated anchor-head scores; fixed heads keep only an on-GPU subset, with the rest permanently discarded. On LibriSpeech-Long with two 3B backbones (Voxtral-mini-3b and Qwen2.5-Omni-3B), WnW preserves near-Full-Cache accuracy while keeping only 20% of audio tokens on GPU, where prefill-only baselines fail to terminate. Results generalize across language, task, and domain shifts, and CPU-GPU recall adds little decode-time overhead in our measurements.

Figures & tables

Figure 1: Attention to audio positions, averaged across

50

samples from the LibriSpeech-Long dev-clean subset (Voxtral). Prefill : from the first answer token; Decode (cumulative) : averaged over all answer tokens. (a) Per-position mass on a uniform grid; (b) cumulative mass. Prefill concentrates near the start (sink effect); decode-cumulative spreads across the full input.

Figure 2: Overview of WnW. Left (Prefill) : KV-heads are partitioned into three roles via offline VS $\times$ HS calibration. Anchor heads retain all audio KV on GPU; tidal heads offload their unselected complement to CPU; fixed heads discard the unselected portion permanently. Right (Decode) : anchor-head attention identifies the top- $k$ audio chunks at each step; tidal heads recall the complement of those chunks from CPU on demand and release it again after $\delta$ consecutive unselected steps (the CPU copy persists). The dashed frame marks one selected chunk, i.e. two segments.

Figure 3: Truncated WER vs. audio KV retention

r_{\text{GPU}}

on LibriSpeech-Long test-clean . Top: Voxtral-mini-3b; bottom: Qwen2.5-Omni-3B. Upward arrows mark points exceeding

30\%

WER (numbers in Table 5 , Appendix). WnW stays close to Full Cache (dashed) at every

r_{\text{GPU}}

; prefill-only baselines degrade sharply below

r_{\text{GPU}}{=}0.6

Method	Category	Retention	clean / other
Full Cache	—	100%	6.79 / 8.86
WnW	KV recall	$\sim$ 20%	6.23 / 8.87
AffPool	token merging	20%	111.63 / 113.25
AffPool	token merging	40%	67.08 / 75.14
AffPool	token merging	60%	10.04 / 11.00
AffPool	token merging	80%	6.47 / 8.94

Table 1: Comparison with a token-merging efficiency baseline on LibriSpeech-Long using Voxtral-mini-3b. Numbers are truncated WER (

\downarrow

, ratio

1.2

		LongSpeech		PriMock57
Method	$r_{\text{GPU}}$	asr-fr $\downarrow$	en2fr $\uparrow$	WER $\downarrow$
Full Cache	100%	20.42	38.48	23.47
Ada-KV	$\sim$ 20%	112.91	2.99	106.43
AudioKV	$\sim$ 20%	105.00	5.22	103.83
ArkVale	$\sim$ 20%	28.63	35.40	32.45
WnW (ours)	$\sim$ 20%	22.68	38.21	24.23

Table 2: Generalization and domain robustness on Voxtral-mini-3b at

r_{\text{GPU}}{=}0.2

(200 samples per LongSpeech split, all

57

PriMock57 consultations, identical for all methods). asr-fr and PriMock57: truncated WER (ratio

1.2

); en2fr: sacreBLEU ( Post, 2018 ) , same truncation. WnW’s measured retention is

r_{\text{GPU}}{=}22.25\%

r_{\text{GPU+CPU}}{=}41.5\%

on LongSpeech and

21.2\%

40.46\%

on PriMock57.

(A1) Voice-head count (WER $\downarrow$ )
$n_{\text{voice}}$	$r_{\text{GPU}}{=}0.1$	$r_{\text{GPU}}{=}0.2$
10	51.92	6.53
30	9.78	6.46
50	8.09	6.39
70	6.31	6.20
90	6.76	6.23

Table 3: Ablations on LibriSpeech-Long test-clean (truncated WER, ratio

1.2

Figure 4: GPU KV cache size over decode steps on a 4-minute LibriSpeech-Long sample (Voxtral-mini-3b). The four curves per method are

r_{\text{GPU}}\in\{0.8,0.6,0.4,0.2\}

. The dotted vertical line marks ground-truth length; methods extending beyond it fail to terminate. WnW’s slight oscillation reflects chunk swap.

Config.	$r_{\text{GPU}}$	ms/token	MB/step
35 tidal heads	21.7%	27.2	0.05
75 tidal heads	21.4%	26.8	0.20
115 tidal heads	21.0%	26.7	0.40
155 tidal heads	20.5%	27.4	0.62
195 tidal heads	20.1%	27.5	0.82
235 tidal heads	19.7%	27.8	1.04

Table 4: Runtime impact of CPU

\to

GPU recall on Voxtral-mini-3b, LibriSpeech-Long test-clean (64 samples) at

\sim

20% GPU retention, on one A100 with batch size

1

. Decode time is median ms/token; MB/step is CPU

\to

GPU KV transfer averaged per decode step.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone: Voxtral-mini-3b
	test-clean (WER $\downarrow$ )				test-other (WER $\downarrow$ )
Method ( $r_{\text{GPU}}{=}$ )	$0.8$	$0.6$	$0.4$	$0.2$	$0.8$	$0.6$	$0.4$	$0.2$
Full Cache	6.79				8.86
Ada-KV	137.46	175.85	196.01	199.24	144.01	181.15	192.60	198.55
AudioKV	6.79	6.31	62.51	192.58	8.85	8.98	66.42	195.53
ArkVale	6.81	6.77	7.12	11.74	8.83	8.85	9.79	14.75

Appendix

Table 5: Full LibriSpeech-Long results across two backbones. Numbers are truncated WER (

\downarrow

, ratio

1.2

). The bottom rows of each backbone block report WnW’s measured on-device retention

r_{\text{GPU}}

and combined retention

r_{\text{GPU+CPU}}

(on-device plus recallable CPU complement).

Method	$r_{\text{GPU}}$	test-clean	test-other
Full Cache	100%	5.70	8.37
WnW (ours)	$\sim$ 20%	11.29	16.63
ArkVale	$\sim$ 20%	15.60	22.20
Ada-KV	$\sim$ 20%	110.51	113.05
AudioKV	$\sim$ 20%	95.02	98.74

Appendix

Table 6: Scaling to Voxtral-Small-24B on LibriSpeech-Long. Numbers are truncated WER (

\downarrow

, ratio

1.2

Chunk size		Anchor count
Value	WER	Value	WER
0.96s	6.45	1	6.22
1.92s	6.29	3	6.23
4.00s (default)	6.23	5 (default)	6.23
5.76s	6.26	7	6.20
7.68s	6.23	10	6.21

Appendix

Table 7: Hyperparameter sensitivity at

\sim

20% GPU retention on Voxtral-mini-3b, LibriSpeech-Long test-clean . Values are truncated WER (

\downarrow

, ratio

1.2

	WER $\downarrow$			Roles kept
Calib.	clean	other	FR	anchor	tidal
LS EN	6.20	8.85	22.43	—	—
LS FR	6.24	8.89	23.55	2/5	76/85

Appendix

Table 8: Cross-dataset calibration robustness on Voxtral-mini-3b. Both configurations use WhisperX threshold

0.85

and target

r_{\text{GPU}}{=}0.2

; numbers are truncated WER (

\downarrow

, ratio

1.2

). The LS EN row is an independent re-run of the default configuration, so it differs marginally from the same configuration in Table 5 . Roles kept reports how many of the anchor and tidal heads chosen by the English calibration are also chosen by the French calibration.

KV-Cache Offloading

KV-cache offloading moves language models’ attention state from GPU memory to CPU, SSD, or remote tiers to expand context and serving capacity.

See all 42 papers

Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
Jul 7, 2026Ke-Han Lu, Keqi Deng, Ruchao Fan +2KV Caching Automatic Speech Recognition
Microsoft, USA · National Taiwan University, Taiwan
AudioKV: KV Cache Eviction in Efficient Large Audio Language Models
Apr 8, 2026Yuxuan Wang, Peize He, Xiyan Gui +5KV-Cache Eviction Audio-Language Models
Xidian University, Xi’an, China · University of Electronic Science and Technology of China, Chengdu, China · Huazhong University of Science and Technology, Wuhan, China +4
VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference
Aug 9, 2026Wenxu Jia, Dongjie Fu, Xize Cheng +6Audio Token Compression LLM Inference Acceleration
1Zhejiang University · 2Meituan

Jul 7, 2026J/K move · Enter open · S save

Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs

Ke-Han Lu, Keqi Deng, Ruchao Fan, Rui Zhao+1

Microsoft, USA · National Taiwan University, Taiwan

KV Caching Automatic Speech Recognition LLM Inference Acceleration KV-Cache Compression Speech Language Models

Open paper AskPDF arXiv

Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length.

Abstract

KV-Cache Offloading

Explore similar work

Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs

AudioKV: KV Cache Eviction in Efficient Large Audio Language Models

VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference