C3: Speech-Guided KV Retrieval for Long-Context SpeechLLM Inference

Dancheng Liu; Markus Müller; Rupak Vignesh Swaminathan; Jing Liu; Srikanth Ronanki; Aram Galstyan

cs.LGOct 30, 2026

C3: Speech-Guided KV Retrieval for Long-Context SpeechLLM Inference

Authors: Dancheng Liu, Markus Müller, Rupak Vignesh Swaminathan, Jing Liu, Srikanth Ronanki, Aram Galstyan

Organizations: University at Buffalo, USA · Amazon AGI, USA

Abstract

Speech language models encode audio at a much higher rate than text, causing the KV cache to dominate the memory, bandwidth, and decoding cost of long-form generation. KV retrieval can reduce per-step access by attending to only a query-dependent subset of the cache, but existing methods are designed primarily for text and do not account for the structure of audio attention. We first show that retrieval errors may compound autoregressively into generation drift. We then analyze full audio attention during long-form generation and identify three properties relevant to retrieval: unequal attention concentration across query heads, non-contiguous audio support, and temporal continuity across decoding steps. Guided by these observations, we propose $C^3$ (Cluster-Compete-Compete), which organizes audio KVs in a hierarchical key-space index, lets query heads sharing a KV head compete for a common retrieval budget, and adaptively reallocates retrieval resolution through paired split-merge updates. Across long-form speech recognition, translation, and summarization on four SpeechLLMs, $C^3$ achieves the strongest overall performance among retrieval methods under comparable final-KV access budgets, with particularly large gains in the tight-budget regime, while improving long-context decoding efficiency and accelerator-memory scaling.

Figures & tables

Figure 1: Overview of C 3 . Audio KVs are organized into a hierarchical key-space index. Query heads sharing a KV head compete over an active frontier for a common retrieval budget, while paired split–merge operations dynamically reallocate frontier resolution.

Qwen3 Omni 30B A3B
	LS-Long( $\downarrow$ )		MLS( $\downarrow$ )		ST( $\uparrow$ )		Summ( $\uparrow$ )
KV Budget	128	512	256	1024	256	1024	256	1024
Full	2.02		4.83		74.75		25.52
Quest	–	193.28	131.87	119.96	5.48	11.52	18.80	22.01
UNIQUE	70.30	2.53	45.20	5.39	21.15	63.68	22.31	24.22
C1 (ClusterKV)	180.94	2.81	75.00	5.62	25.91	70.15	24.06	24.91

Table 1: Long-form speech performance under tight and large audio KV retrieval budgets. Best retrieval results within each model and budget are shown in bold. Results at intermediate budgets and additional ablations are provided in Appendix G.1 .

Panel of Figure 2 — Figure 2: (a) C 3 achieves lossless passcode recovery at substantially smaller retrieval budgets than the baselines. (b) End-to-end latency for audio contexts up to 32K tokens with a 512-KV retrieval budget; retrieval becomes more efficient than full attention as the audio context grows.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Retrieval-induced generation drift. The retrieval trajectory initially follows the relevant audio region but later jumps forward, causing the transcript to lose alignment and terminate prematurely. Curves are smoothed and outliers removed for visualization.

Figure 4: Attention specialization across KV and query heads. Query heads tracking task-relevant acoustic information concentrate on similar regions within each KV group, whereas heads with different preferences exhibit more diffuse attention.

Figure 5: Full audio attention during long-form decoding. (a) The dominant attention band advances with generation; red points mark top- $64$ positions outside the band. (b) Attention near the frontier is itself non-contiguous.

Qwen3 Omni 30B A3B
	LS-Long			MLS			ST			Summ
KV Budget	128	256	512	256	512	1024	256	512	1024	256	512	1024
Full	2.02			4.83			74.75			25.52
Quest	-	172.78	193.28	131.87	146.12	119.96	5.48	8.22	11.52	18.8	20.2	22.01
Quest/mean	178.89	85.1	28.95	100.17	60.03	19.26	9.33	20.61	45.94	20.43	21.65	22.96
+C2	70.33	20.19	3.12	46.06	12.91	5.54	20.68	46.93	62.9	21.94	23.18	24.14

Appendix

Table 2: Long-form speech performance under different audio KV retrieval budgets. We report WER (

\downarrow

) for LibriSpeech-Long (LS-Long) and multilingual long-form ASR (MLS), chrF (

\uparrow

) for speech translation (ST), and ROUGE-L (

\uparrow

) for summarization (Summ). Best results within each model and budget are shown in bold. Dash (–) for Quest indicates that Quest does not support the target budget.

LS-Long
	Qwen3-Omni 30B A3B					Qwen2.5-Omni 3B
EMA $\lambda$	0.0	0.2	0.5	0.8	1.0	0.0	0.2	0.5	0.8	1.0
$R{=}1$	2.31	2.27	2.31	2.22	2.46	15.79	14.88	17.10	18.81	14.31
$R{=}2$	2.04	2.21	1.98	2.12	2.55	12.24	11.09	11.08	10.31	16.89
$R{=}4$	2.07	2.07	2.12	2.07	2.55	9.76	8.75	8.88	9.54	16.89
MLS

Appendix

Table 3: EMA strength

\lambda

(Eq. 2 ) and adaptation rate

R

(paired split–merge exchanges per decoding step). WER (

\downarrow

) at a 256-KV budget with

g{=}16

. The shared default configuration (

\lambda{=}0.5

R{=}2

) is shaded; the best value per model and task block is in bold .

Qwen3 Omni 30B A3B
	AudioMarathon		MMAU-Pro		MSP/acc		MSP/F1
KV	128	256	128	256	128	256	128	256
Full	83.3		61.8		57.9		37.2
Quest	–	79.2	–	57.9	–	54.8	–	33.1
Quest/mean	78.1	80.0	57.0	59.3	53.9	56.4	30.4	34.7
+ C2	79.9	81.3	58.9	59.9	57.3	57.8	36.1	37.0

Appendix

Table 4: Understanding/QA performance under audio KV retrieval budgets, on AudioMarathon (6-subset average accuracy, %), MMAU-Pro (accuracy, %), and MSP-Podcast emotion (MSP/acc: accuracy, %; MSP/F1: macro-F1, %). Higher is better. Best retrieval result within each model and column is in bold.

Panel of Figure 6 — Figure 6: Needle-in-a-haystack performance on Qwen2.5-Omni-3B, Gemma-4, and Voxtral-Mini-3B. Each row reports exact-match and per-digit accuracy as a function of the audio KV retrieval budget.

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 323 papers

WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs
Aug 24, 2026Yiming Yao, Chenyang Lyu, Xuanfan Ni +4KV-Cache Offloading KV-Cache Compression
School of Informatics, Xiamen University · Alibaba Group
Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
Jul 7, 2026Ke-Han Lu, Keqi Deng, Ruchao Fan +2KV Caching Automatic Speech Recognition
Microsoft, USA · National Taiwan University, Taiwan
AudioKV: KV Cache Eviction in Efficient Large Audio Language Models
Apr 8, 2026Yuxuan Wang, Peize He, Xiyan Gui +5KV-Cache Eviction Audio-Language Models
Xidian University, Xi’an, China · University of Electronic Science and Technology of China, Chengdu, China · Huazhong University of Science and Technology, Wuhan, China +4

Aug 24, 2026J/K move · Enter open · S save

WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

Yiming Yao, Chenyang Lyu, Xuanfan Ni, Longyue Wang+3

School of Informatics, Xiamen University · Alibaba Group

KV-Cache Offloading KV-Cache Compression LLM Inference Acceleration Speech Language Models Long-Context Language Modeling

Open paper AskPDF arXiv

Long-form audio inputs make the KV cache the dominant memory cost of speech LLMs. Prefill-only KV compression methods permanently discard audio KV positions once evicted, with no pathway to recover them during decoding. We show this is fragile on long-form audio: prefill attention concentrates near the audio start (an attention-sink effect), while decode-time attention distributes broadly, and the two rankings overlap weakly. We propose WnW (Waxing-and-Waning KV cache), which classifies KV-heads into anchor, tidal, and fixed roles via offline calibration. Anchor heads keep all audio KV on GPU and yield a decode-time signal of which audio region each token is read from; tidal heads keep a CPU-resident complement that is recalled chunk-by-chunk based on aggregated anchor-head scores; fixed heads keep only an on-GPU subset, with the rest permanently discarded. On LibriSpeech-Long with two 3B backbones (Voxtral-mini-3b and Qwen2.5-Omni-3B), WnW preserves near-Full-Cache accuracy while keeping only 20% of audio tokens on GPU, where prefill-only baselines fail to terminate. Results generalize across language, task, and domain shifts, and CPU-GPU recall adds little decode-time overhead in our measurements.

Abstract

Speech Language Models

Explore similar work

WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs

AudioKV: KV Cache Eviction in Efficient Large Audio Language Models