Audio Token Attention Is Predictable Before the Language Model Runs

Kyoungjun Park; Yunzhe Li; Lili Qiu

cs.SDSep 30, 2026

Audio Token Attention Is Predictable Before the Language Model Runs

Authors: Kyoungjun Park, Yunzhe Li, Lili Qiu

Organizations: The University of Texas at Austin

Abstract

A large audio language model (LALM) turns a minute of speech into 750-1,500 tokens and prefills every one. Image-token pruning often cuts after the language model's first layers, where image tokens draw little attention. Audio tokens draw much more attention there, and their ranking is still far from final, so audio needs a ranking before the language model runs. Surprisingly, the attention an audio token will receive across the language model is already linearly predictable from its encoder output, before the language model runs. A linear map, fitted in closed form without labels, predicts this all-layer attention ranking at $ρ\geq .69$ on eleven of thirteen LALMs. Our method, Triage, cuts audio tokens by this prediction and, on multiple choice, cuts again at layer 2, correcting the prediction with the attention observed there. Triage sets its compression without labels, under two budgets that limit how far its output may differ from the model's own full-audio output. At the conservative budget, its word error rate and accuracy stay within .04 of full audio. At the aggressive budget, Triage beats every baseline in all twelve transcription cases. On multiple choice, at 2.2-5x compression, it outperforms DART, the strongest baseline on average, by .043 in mean accuracy. Because it cuts before the language model, it raises the audio that fits in Qwen2.5-Omni-3B's context window from 21.8 to about 62 minutes. At its most compressive point, Triage lets one GPU serve 4x as many concurrent 5-minute streams of that model. Project page: https://audio-triage.github.io

Figures & tables

Panel of Figure 1 — Figure 1: Before the language model runs, the prior predicts which audio tokens it will attend to, but other signals available at that point do so only weakly. All panels: Qwen 30B. (1) A held-out LibriSpeech clip with the median $\rho$ : each audio token’s language-model attention and the prior’s prediction of it. (2) At layer 2, each modality’s share of the text positions’ attention and how well the tokens’ ranking by the answer’s attention through that layer agrees with their all-layer ranking. The shading is schematic. (3) Language-model attention against the prior (green) and four other signals available before the language model, over held-out LibriSpeech tokens ( $\rho$ pooled over all tokens).

$\rho$ against $g$	the prior	enc. self-attn	acoustic energy	state norm $\\|\mathbf{e}_{i}\\|_{2}$	self-information
Ultravox, $1$ B	$\mathbf{+.83}$	$+.17$	$+.28$	$-.45$	$-.26$
Phi-4, $5.6$ B	$\mathbf{+.79}$	$+.23$	$+.12$	$+.03$	$+.12$
Voxtral, $3$ B	$\mathbf{+.79}$	$+.10$	$+.15$	$-.36$	$-.13$
Qwen2.5-Omni, $3$ B	$\mathbf{+.75}$	$+.17$	$+.27$	$-.49$	$-.12$
Qwen2.5-Omni, $7$ B	$\mathbf{+.73}$	$+.18$	$+.29$	$-.54$	$-.27$
Qwen3-Omni, $30$ B	$\mathbf{+.70}$	$+.31$	$+.19$	$-.33$	$-.14$

Table 1: The prior predicts language-model attention on all seven models, while three signals used by existing methods predict it weakly and the state norm changes sign across models. Each entry is the per-clip Spearman

\rho

against

g

, averaged over

100

held-out LibriSpeech clips per model. Fig. 3 b adds three more LALMs (

\rho{=}.82

.88

), DiVA (

\rho{=}.91

) and the two Qwen-Audio models, where the prior fails. DiVA is left out here because its learned query tokens have no time order. Bold : each row’s largest

|\rho|

Figure 2: Where Stage 2 cuts, the prior ranks the audio tokens better than the language model’s own attention, and the two combined rank them best. Qwen 30B, AudioMarathon-RACE. Every ranking is scored against the answer’s attention: the attention the answer pays each audio token over all

48

layers. The blue curve ranks the tokens by that attention summed only through each layer, so it reaches

1

at layer 48 by construction. The green line is the prior, fixed before layer 1. The diamond averages the two rankings at the cut. Past the strip marked cut , the prefill over every token is paid.

Panel of Figure 3 — Figure 3: Deleting the prior’s top tokens costs the most, and one forward pass separates the prior’s failures. (a) Voxtral, LibriSpeech. Each line deletes the same number of tokens by a different rule. Same energy and position delete other tokens that match the prior’s top tokens in acoustic energy or in their spread over the clip. The three controls not drawn (random, the prior’s lowest-ranked and the loudest tokens) cost less at every rate. (b) Thirteen LALMs [ 24 ; 25 ; 27 ; 28 ; 53 ; 51 ; 50 ; 52 ; 26 ; 63 ; 64 ; 65 ] . Each is screened with one forward pass over $20$ unlabelled clips.

Figure 4: Triage: two cuts, one prior. Stage 1 cuts the encoder’s

N

audio tokens to

K_{1}

before the language model runs. The bars above the first strip are the prior

s

, solid where Stage 1 keeps the token. Stage 2 cuts

K_{1}{\to}K

at layer 2, correcting the prior with the attention observed there. Transcription runs Stage 1 alone. Lower left (schematic): top-

K

, used on multiple choice, can leave whole bins with no token, while bin coverage , used on transcription, keeps one token from each of

K_{1}

equal time bins.

	Qwen2.5-Omni-3B			Qwen3-Omni-30B			Voxtral			Phi-4
(a) WER	Libri	FLEURS	TED	Libri	FLEURS	TED	Libri	FLEURS	TED	Libri	FLEURS	TED
full audio	.083	.153	.215	.011	.037	.020	.020	.043	.032	.017	.045	.085
[2pt/1.5pt] conservative ( $\times$ )	1.54	1.11	1.25	1.54	1.54	1.67	1.54	1.54	2.00	1.54	1.54	1.54
ours	.099	.177	.197	.025	.050	.049	.025	.043	.071	.045	.064	.102
[2pt/1.5pt] aggressive ( $\times$ )	2.00	1.18	1.67	2.00	2.00	2.00	2.86	2.86	2.38	2.86	2.86	2.86
uniform pooling	.348	.325	.403	.160	.140	.123	.398	.385	.241	.383	.368	.488

Table 2: Both tasks at both budgets (full results in App. H ). Every selector in a column keeps the same number of tokens,

K{=}N/c

, where

c

is the compression in the first row of each budget. (a) Transcription, corpus WER (lower is better;

n{=}300

per case,

100

on Qwen 30B, seven talks on TEDLIUM). Ours is bin coverage. (b) Multiple choice, accuracy (higher is better;

n{=}400

per case; RACE is AudioMarathon-RACE). Ours is two-stage Triage with precision fusion. Bin coverage is the same one-stage cut as ours in (a) and never sees the question. † Ahead of DART in a one-sided exact McNemar test, Holm-corrected over the twelve cases. Bold : the best selector at the aggressive budget.

Figure 5: At the same

K

, Triage stays within two points of full audio to higher compression than every baseline. Qwen 3B on a second sample of clips (

n{=}400

per benchmark,

241

on RACE), disjoint from Tab. 2 ’s clips on MMSU and DREAM. The red dashed line is full audio, and the shaded area is the two-point band. Triage is inside the band to

2.9\times

on MMSU and DREAM and through

6.7\times

, the highest ratio swept, on RACE. No baseline is inside it past

2\times

on MMSU and DREAM or past

4\times

on RACE.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B.1: Attention mass by layer, the panel behind the attention condition of § 3.1 . Each curve is the share of the text positions’ attention that goes to one modality, at each of the first eight language-model layers (Qwen 30B,

n{=}100

; audio from RACE, images from MMMU, video from MVBench). Audio draws more attention than both visual modalities at every one of these layers. From layer 2 onward, video draws much less attention than audio, closer to images, although it has more tokens than audio and is also temporal.

Model	the passage	tokens	mass at $L{=}2$	mass / token	answered right
Qwen3-Omni-30B	heard	$1{,}170$	$10.16\pm.14$	$.0087\%$	$32/40$
( $n{=}40$ )	read, as a page	$1{,}080$	$\phantom{0}4.36\pm.06$	$.0041\%$	$31/40$
	read, as text	$\phantom{00}318$	$25.87\pm.23$	$.0806\%$	$33/40$
Qwen2.5-Omni-3B	heard	$2{,}250$	$23.17\pm.24$	$.0103\%$	$29/40$
( $n{=}40$ )	read, as a page	$1{,}426$	$10.80\pm.13$	$.0077\%$	$30/40$
	read, as text	$\phantom{00}318$	$25.03\pm.23$	$.0780\%$	$30/40$

Appendix

Table B.1: One passage, three channels, same question. Mass is the attention mass at layer 2, where Stage 2 cuts, as a mean over passages

\pm

its standard error. Tokens are the median over passages, and mass / token is the mean of each passage’s own mass per token. Every

\times

ratio in the text is a per-passage median. Each model answers about as many questions right from the page as from the audio, so both are channels it can use. Phi-4 is not run on text: its two modalities take different LoRA adapters, and plain text takes neither. This run uses its own prompt, so its masses can be compared across its rows but not with the

16.2\%

of § 3.1 .

audio tokens kept	$65\%$	$50\%$	$35\%$
linear prior	$.139$	$.186$	$.348$
MLP prior	$\mathbf{.092}$	$\mathbf{.126}$	$\mathbf{.195}$
best baseline	$.101$ ( vad )	$.144$ ( dart )	$.285$ ( dart )
true attention (oracle)	$.094$	$.111$	$.195$

Appendix

Table C.1: On Qwen2-Audio, an MLP in the prior’s place gives Stage 1 the lowest WER of any deployable score. The model is Qwen2-Audio-7B-Instruct on LibriSpeech (

n{=}200

), where full audio has WER

.083

. Each row gives the WER of Stage 1’s bin coverage with one score, or of the best of six baselines at each budget ( pool , vad , random , dart , fastv and FastV’s all-rows variant). True attention is bin coverage on the language model’s own

g

, an oracle that needs the full prefill. Bold marks the lowest WER without the oracle.

Model (tok/s)	audio	$N$	full (ms)	$.65$	$.50$	$.35$
Qwen2.5-Omni-3B ( $25$ )	$5$ min	$7{,}500$	$315.2$	$1.61\times$	$2.06\times$	$\mathbf{2.92\times}$
Qwen2.5-Omni-3B ( $25$ )	$10$ min	$15{,}000$	$695.1$	$1.65\times$	$2.19\times$	$3.16\times$
Voxtral-Mini-3B ( $12.5$ )	$5$ min	$3{,}750$	$194.5$	$1.56\times$	$2.01\times$	$2.71\times$
Voxtral-Mini-3B ( $12.5$ )	$10$ min	$7{,}500$	$406.6$	$1.59\times$	$2.08\times$	$\mathbf{2.98\times}$
Phi-4-multimodal ( $12.5$ )	$5$ min	$3{,}750$	$267.8$	$1.46\times$	$1.86\times$	$2.08\times$
Phi-4-multimodal ( $12.5$ )	$10$ min	$7{,}500$	$540.7$	$1.56\times$	$2.02\times$	$\mathbf{2.78\times}$

Appendix

Table C.2: At a matched

N{=}7{,}500

audio tokens, keeping

35\%

speeds prefill up

2.8

–

3.0\times

on all three models. We measure prefill latency (A100-80GB, bf16, sdpa on all three) on synthetic audio of the stated duration, for full audio and for keeping

65/50/35\%

of the audio tokens. Each latency is a mean over timed passes after two warm-up passes. “

\times

” is full

\,\div\,

kept. Voxtral and Phi-4 emit

12.5

tok/s, and Qwen 3B emits

25

tok/s. Bold marks the entry that keeps

35\%

in the matched

N{=}7{,}500

row of each block. Absolute latencies differ with language-model width and depth.

		before fitting	after fitting
model	audio front end	$\Delta_{2}$	$\rho$
above the $0.25$ threshold: the prior works
Qwen3-Omni-30B (MoE)	trained together	$2.100$	$.701$
SeaLLMs-Audio-7B [ 63 ]	Qwen2-Audio $\to$ Qwen2.5	$1.342$	$.819$
Aero-1-Audio [ 65 ]	Qwen2-Audio $\to$ Qwen2.5	$1.143$	$.879$
Ultravox-v0.5-Llama-3.2-1B [ 53 ]	Whisper $\to$ Llama-3.2	$0.941$	$.829$

Appendix

Table C.3: A threshold of

0.25

\Delta_{2}

splits the thirteen models.

\Delta_{2}=\lVert\mathbf{h}_{2}-\mathbf{h}_{0}\rVert/\lVert\mathbf{h}_{0}\rVert

is the relative distance the audio positions move over the language model’s first two layers, from one forward pass over

20

unlabelled clips. Nine of the thirteen start from a Whisper encoder and fall on both sides, so the encoder family alone does not decide the side. Above the threshold,

\Delta_{2}

does not rank

\rho

: it returns a side, not a distance. Bold : the highest

\rho

Model	benchmark	prior	obs.
Qwen 3B	MMSU	.678	.516
	DREAM	.644	.467
	RACE	.564	.497
Qwen 30B	MMSU	.574	.414
	DREAM	.577	.401
	RACE	.548	.391

Appendix

Table C.4: The encoder-side prior remains necessary at Stage 2. Each entry is the token-level top-

g

recall when keeping

20\%

of the tokens. The prior retrieves the top tokens by converged

g

better than raw cumulative layer-2 observation ( obs. ) in all six multiple-choice cases, by

+.067

+.177

, with five of the six gaps above

+.15

. Each benchmark uses up to

150

clips.

	Qwen2.5-Omni-3B			Qwen3-Omni-30B (MoE)
Benchmark	in-dom.	univ.	0-shot	in-dom.	univ.	0-shot
speech QA
MMSU	.698 $\pm$ .006	.759 $\pm$ .003 ∗∗	.756 $\pm$ .004	.653 $\pm$ .007 ∗∗	.629 $\pm$ .007	.613 $\pm$ .007
AudioM.-RACE	.666 $\pm$ .002	.713 $\pm$ .002 ∗∗	.690 $\pm$ .002	.561 $\pm$ .003	.559 $\pm$ .003	.549 $\pm$ .003
DREAM	.675 $\pm$ .004	.745 $\pm$ .003 ∗∗	.740 $\pm$ .003	.646 $\pm$ .005	.645 $\pm$ .005	.635 $\pm$ .005
transcription

Appendix

Table C.5: What draws language-model attention is largely universal, not task-specific (

n{=}200

per benchmark). Each entry is the Spearman

\rho

between predicted and measured language-model attention

g

, for a per-task prior ( in-dom. ), one universal prior trained on a pooled mix of all six benchmarks, or a 0-shot prior trained on all other benchmarks and tested on the held-out one. Bold marks the better of in-dom./univ. where the pair differs ( ∗∗ : beyond

2

SE and

{\geq}.01

) and both where it does not. On Qwen 3B the universal prior leads the per-task one on all six benchmarks, significantly on four, and the 0-shot prior is at most

.027

below the universal one. On Qwen 30B the per-task prior keeps an edge on MMSU and LibriSpeech, the universal prior has a far larger edge on MMAU-mini, and the 0-shot prior is at most

.021

below the universal one.

Condition	fwd	tokens	20	25	30	35	40	45 min	mean
oracle span (ceiling)	1	—	.847	.857	.873	.803	.800	.843	.837
chunked, full audio	1	$N$	.730	fails: every question over context					—
single call (today’s default)	1	$N$	.513	.500	.497	.410	.470	.500	.482
per-minute window voting	$D$	$N$	.700	.717	.740	.667	.677	.733	.706
Triage, conserv. (.50/.35)	1	$.35N$	.787	.773	.780	.630	.700	.713	.731
Triage, aggr. (.35/.20)	1	$.20N$	.733	.763	.787	.607	.697	.743	.722

Appendix

Table C.6: Long audio: what runs, and at what cost (Qwen 3B, needle-in-a-haystack QA over concatenated RACE articles, accuracy

\uparrow

). Each entry averages

100

questions at each of three needle positions, and the six durations hold

1{,}800

questions. The positions are

15

50

and

85\%

of the stream.

D

is the nominal duration in minutes and

N

the encoder tokens, about

1{,}500D

and fewer where a stream runs short of

D

. Two rows are failure modes. Chunked full audio overflows the context on every question past

{\approx}21.8

minutes, and the single call, today’s default, silently truncates . The informative baseline is per-minute voting, the only other condition that both runs and answers the question asked. Triage runs at the two operating points the label-free calibration picked at

20

minutes, kept unchanged at every duration. Its shortlists are

.50N

and

.35N

. A shortlist wider than the context window is clamped to the positions the prompt leaves free, which at

45

minutes trims the conservative one on

282

300

streams. The last row uses a prior fit on

40

RACE clips from outside the article pool the streams are built from, and is the selector run at

80

minutes. Bold marks the best method at each duration, not counting the oracle. The mean column is not bolded.

$20$ min, all $k$ needles required	tokens	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}4$
full audio (uncompressed reference)	$N$	$.71$	$.52$	$.40$	$.38$
Triage, conserv. (.50/.35)	$.35N$	$\mathbf{.78}$	$\mathbf{.62}$	$.48$	$\mathbf{.40}$
Triage, aggr. (.35/.20)	$.20N$	$.72$	$.56$	$\mathbf{.51}$	$.34$
Stage 1 alone
universal prior	$.35N$	$.73$	$\mathbf{.62}$	$.49$	$.37$
universal prior	$.25N$	$.74$	$.57$	$.36$	$.29$

Appendix

Table C.7:

k

answer-bearing articles spread through one

20

-minute stream (

n{=}100

per

k

). An item counts as right only when all

k

answers are right, and the stream stays below the context limit, so full audio still runs. The tokens column gives how many tokens each condition keeps in the end. Triage runs the two operating points of Tab. C.6 . Bold marks the best selector in each column. Full audio is the uncompressed reference and is not one of the selectors.

	conservative		aggressive
Method	acc	$\Delta$ [95% CI]	acc	$\Delta$ [95% CI]
(a) Qwen3-Omni-30B, $r_{1}/r_{2}$ of $.85/.55$ and $.65/.35$ , reference full audio
full audio (reference)	$.779$
no audio	$.428$ $-.351$ [ $-.390,-.313$ ]
one-minute window vote	$.667$ $-.112$ [ $-.142,-.084$ ]
pool	$.708$	$-.071$ [ $-.097,-.044$ ]	$.476$	$-.303$ [ $-.345,-.261$ ]

Appendix

Table C.8: Natural long recordings, main set (QuALITY stories read by LibriVox volunteers,

20

–

43

minutes, four-option multiple choice, accuracy

\uparrow

\Delta

is paired against each panel’s reference, with a

95\%

interval from a bootstrap that resamples stories. Qwen 30B answers

815

questions in

90

stories, and Qwen 3B answers all

1{,}590

. Qwen 3B’s conservative point overflows its context on the longest stories, so that column covers

56

stories (

999

questions, on which truncation scores

.527

), and on

40

of them its shortlist is trimmed to fit. Here fastv ranks all

N

tokens by the attention that Stage 2 reads and, like FastV, continues the same forward pass after the cut. On Qwen 3B, it needs the whole recording in context, so it is run on the same truncated recording as the reference. Every row under an operating point keeps the same number of tokens.

	Qwen 30B, conservative		Qwen 30B, aggressive		Qwen 3B, aggressive
Method	acc	$\Delta$ [95% CI]	acc	$\Delta$ [95% CI]	acc	$\Delta$ [95% CI]
reference	$.703$				$.560$
no audio	$.484$ $-.220$ [ $-.293,-.141$ ]				$.363$	$-.197$ [ $-.264,-.130$ ]
one-minute window vote	$.703$ $0$ [ $-.058,+.056$ ]				$.591$	$+.031$ [ $-.021,+.082$ ]
pool	$.679$	$-.024$ [ $-.070,+.021$ ]	$.508$	$-.195$ [ $-.265,-.127$ ]	$.456$	$-.104$ [ $-.147,-.062$ ]
vad	$.715$	$+.012$ [ $-.045,+.072$ ]	$.598$	$-.106$ [ $-.173,-.041$ ]	$.519$	$-.041$ [ $-.092,+.008$ ]

Appendix

Table C.9: Natural long recordings, continuation set (

27

of the stories extended with the reader’s narration of the rest of the story,

45.6

60

minutes). The questions are the main set’s, and none asks about the added narration. Rows are as in Tab. C.8 .

\Delta

is paired against full audio on Qwen 30B and against truncation on Qwen 3B. Qwen 30B answers

246

questions, and Qwen 3B answers

482

. On Qwen 3B only the aggressive point fits, and fastv does not run. The last block compares the reference and deployed Triage with the main set, on the same questions. Truncation never reaches the added narration, so its difference is zero by construction.

Model	Method	encoder	language-model prefill	end to end
Qwen 30B	no audio	—	$2.26$	$2.26$
	full audio (reference)	$0.85$	$5.54$	$6.39$
	one-minute window vote	$0.85$	$7.43$	$8.31$
	Triage, Stage 1 only: top- $K$ , conservative	$0.85$	$4.02$	$4.84$
	Triage, both stages, conservative	$0.84$	$4.09$	$4.96$
	Triage, Stage 1 only: top- $K$ , aggressive	$0.85$	$3.51$	$4.37$

Appendix

Table C.10: Where the time goes on the main set (median seconds per question on one A100, timed cold : nothing is cached across questions or methods, though the weights stay loaded and discarded runs warm the GPU first; batch

1

, except that window voting batches its windows). Each column is a median taken separately over the timed questions, so the columns need not add up. Qwen 3B’s conservative rows cover the stories where that point fits.

Signal	MMSU	DREAM	RACE	MMSU, 2nd sample
Acoustic energy	$+.236$	$+.216$	$+.115$	$+.276$
Self-information ( $k{=}4$ )	$+.059$	$-.080$	$-.150$	$-.191$
State norm $\\|\mathbf{e}_{i}\\|_{2}$	$-.346$	$-.514$	$-.538$	$-.370$

Appendix

Table D.1: Three free signals are scored against language-model attention on each benchmark. Each entry is the Spearman

\rho

of a signal against

g

on Qwen 3B. Self-information is the residual of a linear autoregressor that predicts each token’s encoder output from the

k

before it. The last column is a second MMSU sample (

120

clips,

23{,}951

tokens), which Tab. D.2 and App. D.2 also use.

Signal	Representation	$\rho$ vs. $g$
The prior (ours)	encoder $\mathbf{e}_{i}$	$\mathbf{+.67}$
Self-information ( $k{=}4$ )	encoder $\mathbf{e}_{i}$	$-.16$
Self-information ( $k{=}4$ )	raw waveform	$+.24$
Self-information ( $k{=}16$ )	raw waveform	$+.11$
Spectral entropy	raw waveform	$+.01$
Acoustic energy	raw waveform	$+.28$

Appendix

Table D.2: However self-information is measured, it stays far below the prior. Each entry is the Spearman

\rho

of a signal against

g

per clip, on the second MMSU sample of Tab. D.1 . Here self-information is the residual of a linear autoregressor fitted on training clips and applied to test clips. The last column of Tab. D.1 fits one autoregressor per clip, which gives

-.191

Bench	Operating point	full	top- $K$ (ours)	self-info	inverse	VAD	random
MMSU	conserv.	.603	.603	.583	.585	.595	.608
MMSU	aggressive	.603	.583	.515	.520	.552	.505
DREAM	conserv.	.894	.861	.582	.801	.831	.747
DREAM	aggressive	.894	.806	.489	.584	.622	.554
RACE	conserv.	.818	.835	.738	.777	.820	.782
RACE	aggressive	.818	.820	.615	.688	.718	.677

Appendix

Table D.3: Prompt compression’s rule, adapted to audio, is at or below random in five of six cases. Each entry is gold accuracy on Qwen 3B at the two deployed operating points (

n{=}400

). The clips are those of Tab. 2 b, except that DREAM uses a separate sample, on which full audio scores

.894

(against

.892

on the grid). self-info keeps the top-

K

tokens by self-information, and inverse keeps the bottom-

K

. Bold marks our selector, which is ahead of every baseline in five of the six cases. In four of them the lead exceeds the

{\pm}.021

SE, and the largest lead is

.184

	LibriSpeech		FLEURS
Selector	aggressive	conservative	aggressive	conservative
Qwen2.5-Omni-3B
kept	$.50$	$.65$	$.85$	$.90$
full audio (ceiling)	$.0827$	$.0827$	$.1528$	$.1528$
bin coverage (ours)	$\mathbf{.1160}$	$\mathbf{.0989}$	$\mathbf{.1738}$	$\mathbf{.1774}$
stride (decimation)	$.3693$	$.2195$	$.1976$	$.1778$

Appendix

Table D.4: Bin coverage has lower WER than decimation, neural VAD and word onsets in every case. Each entry is corpus WER ( lower is better ) on the clips of Tab. H.4 , with

n{=}300

per Qwen 3B case and

100

per Qwen 30B case. Selectors shared with that table score the same here, and each model runs at its own operating points for the two budgets. stride keeps

K

evenly spaced frames and discards the rest. vadnn is the released Silero VAD [ 54 ] , and wordonset keeps the frames nearest a word onset in a greedy CTC decode by wav2vec 2.0 ( base-960h ) [ 30 ] , so neither uses a reference. Bold marks the best selector in each column.

(a) Retained set, $10\%$ kept
Metric	Ours	attn top- $K$	Random
Silence fraction (TEDLIUM)	.018	.088	.154
Function-word share (LibriSpeech)	.144	.190	.291
Mean RMS energy	.091	.074	.060
(b) What each stage drops ( $z$ -scored)
Removed at	prior	observed	all-layer

Appendix

Table E.1: The prior keeps the least silence and the fewest function words, and each stage drops a different kind of token. (a) Composition of the retained set at a kept fraction of

10\%

, against attention top-

K

and random. Attention top-

K

keeps the tokens the language model itself attends to most, summed over its layers and heads. Bold marks the best value on each metric. (b) Every audio token, grouped by the stage that drops it (

z

-scored within clip; RACE,

n{=}60

). In this panel Stage 2 reads only the last eight prompt rows. Stage 1 drops low-prior tokens. Stage 2 drops tokens that the prior does not rank low but that receive low observed attention ( bold ). The kept set ranks highest in every column, including the language model’s all-layer attention. Stage 1 never uses that attention, and Stage 2 sees only its first two layers.

$\rho$ between the prior and $g$	on LibriSpeech	on FLEURS
no controls	$.755$	$.770$
controlling for waveform descriptors	$.741$	$.741$
(position, RMS, spectral centroid, flatness, rolloff, zero-crossing rate, onset strength)
controlling for lexical descriptors	$.740$	$.747$
(in-word, content-word, distance to a word boundary, word duration)
controlling for both	$\mathbf{.735}$	$\mathbf{.733}$

Appendix

Table E.2: The prior keeps nearly all of its

\rho

when waveform and lexical descriptors are controlled for (

\rho

between the prior and

g

, so higher is better ;

150

clips to fit and

100

to test). Each row removes from both the prior and

g

everything that the listed descriptors explain. Nothing is removed from the audio. If the prior were a linear readout of loudness and position, the bold row, with both families removed at once, would be zero. The last row gives the bold row as a share of the uncontrolled

\rho

	Voxtral-Mini-3B ( $30$ layers)		Qwen2.5-Omni-3B ( $36$ layers)
$g$ summed over	share of $g$	$\rho$	share of $g$	$\rho$
every layer	$100\%$	$\mathbf{.821}$	$100\%$	$\mathbf{.764}$
layer $3$ and deeper	$93\%$	$.816$	$88\%$	$.759$
layer $9$ and deeper	$72\%$	$.794$	$74\%$	$.751$
the deepest quarter	$21\%$	$.784$	$22\%$	$.710$
layer $1$ alone	$3\%$	$.474$	$6\%$	$.491$

Appendix

Table E.3: The prior predicts attention in the deepest layers too, and does worst on layer 1 alone (

\rho

against

g

restricted to a layer range, with the prior refit to each range;

120

clips to fit,

100

to test, in a run separate from Tab. 1 ’s). If the prior only recovered layer 1’s bilinear form from

\mathbf{e}

, the row for layer 1 alone would have the highest

\rho

here and the deep ranges would collapse. Neither happens. Bold : the all-layer target the prior is fit against.

Control (same #tokens removed)	delete $10\%$	delete $20\%$	delete $35\%$
Voxtral-Mini-3B, WER
at random	$+.510$ [ $+.440,+.580$ ]	$+.738$ [ $+.666,+.813$ ]	$+.659$ [ $+.541,+.741$ ]
the prior’s own bottom	$+.544$ [ $+.471,+.617$ ]	$+.851$ [ $+.788,+.918$ ]	$+.911$ [ $+.879,+.941$ ]
the highest-energy frames	$+.461$ [ $+.391,+.532$ ]	$+.632$ [ $+.533,+.719$ ]	$+.542$ [ $+.445,+.608$ ]
matched to its position	$+.421$ [ $+.347,+.497$ ]	$+.754$ [ $+.693,+.817$ ]	$+.860$ [ $+.819,+.901$ ]
self-information	$+.391$ [ $+.318,+.465$ ]	$+.259$ [ $+.050,+.427$ ]	$+.007$ [ $-.075,+.082$ ]

Appendix

Table E.4: Deleting the prior’s top tokens costs more than every control, on Voxtral and on Qwen 3B. Each cell is prior-top deletion minus control deletion, with a

95\%

bootstrap interval over the same clips. Top: the margins of Fig. 3(a) , on Voxtral-Mini-3B and LibriSpeech (

n{=}100

), in mean per-clip WER. Voxtral’s prior is fit for this intervention on

120

other LibriSpeech clips. Seventeen of its eighteen margins exclude zero. The exception is self-information at

35\%

, where both deletions have saturated (

.935

against

.927

WER, full audio

.070

). Bottom: Qwen 3B on MMSU, DREAM and RACE (

150

clips each), in gold-answer margin (the gold option’s logit minus the best other option’s), entered as its drop from full audio. This run uses the deployed prior unchanged (

\rho{=}.71

.75

on its clips). Sixteen of its eighteen margins exclude zero, and both exceptions are against the highest-energy frames. Bold : the energy-matched control, the one the claim depends on.

Bench	kept $r_{2}$	full	obs tail	obs	prior	precision fusion
MMSU	$35\%$	$.603$	$.560$	$.570$	$.580$	$\mathbf{.585}$
	$20\%$		$.535$	$.573$	$\mathbf{.583}$	$.580$
	$10\%$		$.507$	$.525$	$.545$	$\mathbf{.550}$
DREAM	$35\%$	$.877$	$.792$	$.828$	$.830$	$\mathbf{.835}$
	$20\%$		$.723$	$.745$	$\mathbf{.797}$	$.777$
	$10\%$		$.600$	$.613$	$.630$	$\mathbf{.665}$

Appendix

Table F.1: At layer 2 the prior selects better than the attention observed there, in eight of the nine settings. Each entry is gold accuracy on Qwen 3B after a cut at layer 2 that keeps the fraction

r_{2}

of audio tokens ranked highest by each signal (

n{=}400

per benchmark). The DREAM and RACE clips differ from those of the multiple-choice grid. Every selector chooses from the full

N

, so only the ranking signal differs, and the prior alone is computed before any language-model layer runs. The obs tail column is FastV’s signal as usually implemented, the attention from the trailing prompt positions. The obs column takes the same attention from every prompt-suffix position, as Eq. 2 does. Bold marks the best selector in each row, which is always the prior or precision fusion.

Figure F.1: The prior’s shortlist makes a cut at layer 2 accurate, and a cut that early retains little KV. (a) plots answer fidelity against cut layer

L

on DREAM at the aggressive point (

r_{1}{=}.5

r_{2}{=}.2

n{=}100

). Answer fidelity is the pruned model’s probability for the option that the full-audio pass picks. FastV, which observes before it cuts and so prefills all

N

, needs about eight layers to reach what our shortlist gives at layer 2, and precision fusion adds to it. (b) plots retained audio KV and prefill against

L

at the same point. Our cut at layer 2 retains about

22\%

of the audio KV. FastV’s cut at layer 8, where it first matches that fidelity, retains

38\%

, and later cuts retain more.

			prior shortlist		random shortlist		effect
Bench	Operating point	full	observed- only	precision fusion	observed- only	precision fusion	$\Delta_{\rm shortlist}$	$\Delta_{\rm rule}$
MMSU	conserv. (.85/.65)	.603	.600	.600	.595	.593	$+.005$	$.000$
MMSU	aggr. (.35/.25)	.603	.580	.585	.548	.538	$+.033$	$+.005$
DREAM	conserv. (.65/.45)	.892	.878	.865	.788	.805	$+.090$	$-.013$
DREAM	aggr. (.50/.20)	.892	.815	.860	.675	.713	$+.140$	$+.045$
RACE	conserv. (.85/.65)	.818	.815	.823	.808	.813	$+.008$	$+.008$

Appendix

Table F.2: Replacing the prior’s shortlist with a random one costs accuracy, most at the aggressive point. Each entry is gold accuracy on Qwen 3B with the universal prior and a cut at layer 2 (

n{=}400

clips per benchmark), at each benchmark’s deployed operating points (

r_{1}

r_{2}

in parentheses). Columns cross two shortlists (the prior’s, or a random one of the same size

K_{1}

) with two survivor rules (observed-only, which is FastV on that shortlist, or precision fusion).

\Delta_{\rm shortlist}

is prior minus random under the observed-only rule, and

\Delta_{\rm rule}

is precision fusion minus observed-only on the prior shortlist, both computed before rounding. The three largest shortlist effects, on DREAM at both points and on RACE at the aggressive one, are

3.4

4.6

times their unpaired standard error. This table comes from a separate run, and its margins are taken within that run. Its entries differ from Tab. H.3 only in three DREAM cells, each by

.003

Design choice	acc
Stage 2, fusion rule on a fixed shortlist (prior only: $.750$ )
observed-only ( $\alpha{=}1$ )	.762
product	.765
uniform blend ( $\alpha{=}.5$ )	.766
max-rank	.767
cascade (multi-layer)	.767

Appendix

Table F.3: Precision fusion is the most accurate of eight survivor rules on a fixed shortlist. Each entry is the mean accuracy of Qwen 3B on the multiple-choice grid (

n{=}400

per benchmark), averaged over benchmarks, budgets and cut layers

L{\in}\{2,8\}

. The shortlist,

L

and

K

are the same in every row, so only the rule changes. The

\alpha{=}1

row ranks by the observed attention alone, which is FastV on that shortlist. Precision fusion fits no parameters, yet it is above five unlearned fusions and a supervised learned gate. Bold marks our rule.

			direct cut		achieved vs direct cut
Benchmark	kept	$n$	SD / SE	$\chi^{2}$ $p$	SD ratio	exact $p$
DREAM	$35\%$	$400$	$1.05$	$.34$	$1.63$	$<.0001$
DREAM	$25\%$	$400$	$1.48$	$.005$	$1.99$	$.0005$
DREAM	$15\%$	$399$	$1.54$	$.002$	$\mathbf{2.25}$	$<.0001$
RACE	$35\%$	$317$	$0.45$	$1.00$	$2.59$	$.042$
RACE	$25\%$	$316$	$0.75$	$.90$	$\mathbf{2.87}$	$.0004$

Appendix

Table F.4: On DREAM and RACE the SD ratio grows as the budget tightens, and only MMSU at

25\%

kept shows no effect. The test covers the sixteen language-model targets of App. F.2 on Qwen 3B. MMSU uses the clips of the multiple-choice grid, and DREAM and RACE use those of the target study. The direct cut keeps the tokens that each target itself ranks highest, and achieved is the accuracy of cutting on the prior fitted to that target. SD / SE puts the spread of direct-cut accuracy across targets in units of the binomial error of a test set that size, and

\chi^{2}

p

tests it against a flat direct cut. The SD ratio divides the achieved spread by the direct-cut spread, with an exact

p

from the swap permutation test. Bold marks the largest SD ratio on each of DREAM and RACE.

Benchmark	What it tests	Metric	Scored in
Multiple-choice QA
MMSU [ 35 ]	targeted listening, ${\sim}6$ s	gold accuracy	Tab. H.3
DREAM [ 36 ; 49 ]	dialogue comprehension, ${\sim}45$ s	gold accuracy	Tab. H.3
AudioMarathon-RACE [ 37 ; 55 ]	reading comprehension, ${\sim}2.5$ min	gold accuracy	Tab. H.3
Transcription
LibriSpeech [ 32 ]	read-aloud speech, test-clean	corpus WER	Tab. 2

Appendix

Table G.1: The nine settings, what each one tests, and where it is scored. The three breadth checks sit outside the deployed grids. Architecture figures quoted in the text are Qwen 3B’s: a frozen audio encoder (

32

layers,

{\approx}25

tokens per second of audio) feeding a

36

-layer language model.

Model	Bench	Triage	DART	layer 2	shortlist
Qwen 3B	MMSU	.585	.547	.550	.565
	DREAM	.863	.757	.738	.778
	RACE	.820	.805	.812	.818
Qwen 30B	MMSU	.682	.620	.652	.618
Qwen 30B	DREAM	.943	.907	.853	.843
Voxtral	MMSU	.525	.532	.537	.448

Appendix

Table G.2: Moving DART into the language model does not close the gap on DREAM (accuracy on the clips and at the aggressive operating points of Tab. H.3 ,

n{=}400

per case). layer 2 runs DART’s rule on the layer-

2

hidden states over all

N

tokens, and shortlist does so over Stage 1’s

K_{1}

tokens. On Qwen 30B, all three DART columns come from an A100 run (App. G.1 ), so they can differ slightly from Tab. H.3 . Bold : the best selector in each row.

Selector	MMSU	DREAM	RACE
Qwen2.5-Omni-3B , two disjoint samples, A / B ( $n{=}400$ ; RACE $n{=}241$ each)
precision fusion (ours)	$\mathbf{4.0}$ / $\mathbf{2.9\times}$	$\mathbf{4.0}$ / $\mathbf{2.9\times}$	$\mathbf{4.0}$ / $\mathbf{\geq 6.7\times}$
top- $K$ (prior only, ours)	$\mathbf{4.0}^{\dagger}$ / $2.0\times$	$1.5$ / $1.5\times$	$2.9$ / $4.0\times$
DART	$2.0$ / $2.0\times$	$2.0$ / $2.0\times$	$2.0$ / $4.0\times$
HeadRouter N	$2.0$ / $2.0\times$	$2.0$ / $2.0\times$	$2.9$ / $2.9\times$
FastV N	$2.0$ / $2.0\times$	$1.5$ / $1.5\times$	$1.5$ / $1.5\times$

Appendix

Table H.1: Iso-quality compression: the largest ratio each selector sustains within two points of full audio. Each selector is swept over five compression ratios,

1.5\times

6.7\times

, with

n{=}400

per case unless a block header says otherwise. On Qwen 3B, sample A is the grid sample of Tab. H.3 and sample B is the next

400

clips of the same pool, so the two are disjoint. RACE’s pool holds only

483

clips, so there A is the first

241

clips of the grid sample and B the next

241

. “—” means the selector is outside the band even at the smallest ratio, and “

\geq 6.7\times

” means it is still inside at the largest. Bold marks the largest ratio in a column, ties included. On Qwen 3B, precision fusion is one to two ratio steps above DART in all six samples. On Qwen 30B, precision fusion or top-

K

sustains

2\times

on each benchmark, and DART does so on DREAM and RACE.

model	bench	ours	dart	$\Delta$	$b$	$c$	Holm
Qwen2.5-Omni-3B	MMSU	$.585$	$.547$	$+.037$	$30$	$15$
	DREAM	$.863$	$.757$	$+.105$	$51$	$9$	✓
	RACE	$.820$	$.805$	$+.015$	$13$	$7$
Qwen3-Omni-30B	MMSU	$.682$	$.620$	$+.062$	$45$	$20$	✓
	DREAM	$.943$	$.902$	$+.040$	$22$	$6$	✓
	RACE	$.828$	$.807$	$+.020$	$27$	$19$

Appendix

Table H.2: Precision fusion leads DART in eleven of the twelve aggressive multiple-choice cases. Ours is precision fusion, and each case has

n{=}400

items.

b

counts the items that precision fusion answers correctly and DART does not, and

c

counts the reverse, so

\Delta=(b-c)/n

, computed before rounding. Significance comes from an exact McNemar test [ 48 ] on the

b+c

items where the two differ, and Holm’s method corrects the one-sided

p

-values across the twelve cases. The Holm column marks the five cases that are significant on their own after this correction. Pooled over all

4{,}800

items, the margin is

+.043

		keep $35\%$		keep $25\%$
Target $g$ : query rows $\mathcal{Q}$	$\rho$	direct cut	achieved	direct cut	achieved
trailing rows	.658	.817	.782	.780	.695
all text rows	.677	.825	.825	.818	.770
text rows, template tail dropped	.675	.827	.823	.810	.772
audio rows (ours)	.707	.815	.825	.798	.815
all rows	.714	.815	.825	.802	.812

Appendix

Table H.6: What should the Stage-1 target be? The study runs Qwen 3B on DREAM (

n{=}400

, a sample of its own rather than the grid’s) at two keep rates within the deployed range. Each row is a candidate definition of

g

. Direct cut is the accuracy of cutting on the target itself,

\rho

is how well the prior predicts the target, and achieved is the accuracy of cutting on the prior fitted to that target. Setting the energy control aside, achieved accuracy is more closely rank-correlated with

\rho

than with the direct cut:

+.91

against

+.64

35\%

kept, and

+.66

against

+.23

25\%

. Bold marks the deployed target. At

25\%

kept it has the highest achieved accuracy, and the top five lie within

.005

of each other (per-case SE

\pm.020

at these accuracies). The trailing-row target has a direct cut close to ours but achieves far less, and on Phi-4 the deletion test of App. E.3 separates the two as well.

		Qwen2.5-Omni-3B ( $5$ min)				Qwen3-Omni-30B (MoE, $40$ min)
Task	Budget	$r_{1}/r_{2}$	prefill / KV%	streams	tok/s	$r_{1}/r_{2}$	prefill / KV%	streams	tok/s
MCQ	conserv.	.65/.45	2.3 $\times$ / $46\%$	2 $\times$	2.1 $\times$	.85/.55	1.4 $\times$ / $56\%$	2 $\times$	1.9 $\times$
MCQ	aggr.	.50/.20	3.7 $\times$ / $22\%$	4 $\times^{\ast}$	4.5 $\times$	.85/.35	1.6 $\times$ / $37\%$	3 $\times$	2.8 $\times$
ASR	conserv.	.85/.65	1.6 $\times$ / $66\%$	1.5 $\times$	1.5 $\times$	.90/.65	1.2 $\times$ / $66\%$	1.5 $\times$	1.5 $\times$
ASR	aggr.	.80/.50	2.0 $\times$ / $52\%$	2 $\times$	1.9 $\times$	.70/.50	1.5 $\times$ / $51\%$	2 $\times$	2.0 $\times$

Appendix

Table I.1: Efficiency at the deployed operating points, against full audio on one

96

GB GH200. Per request, we report the language-model prefill speedup and the share of the audio KV cache that is kept (KV%). We also report the largest number of concurrent streams that fit in memory and their aggregate decode throughput (tok/s), timed on one decoding step at that batch size. Prefill excludes the audio encoder, which both sides run on every frame and which holds no KV cache. Each row is the most compressible multiple-choice (MCQ) or short-utterance transcription (ASR) point at its budget: DREAM on Qwen 3B and MMSU on Qwen 30B for MCQ, and LibriSpeech for ASR. The ASR rows time both stages at the transcription points. Transcription itself deploys a single cut to

r_{2}

before the language model, which passes

K

rather than

K_{1}

tokens through the first two layers, so it saves at least as much. Bold marks the aggressive rows. ∗ The batch search stops at

1024

streams, and this point reaches it.

Config	prefill	KV%	streams	tok/s
Qwen2.5-Omni-3B (5-min), aggressive (MCQ) $.50/.20$
full ( $N$ )	$120$ ms	$100\%$	256	666
Stage-1 only ( $K_{1}$ )	$57$ ms ( $2.1\times$ )	$50\%$	512	1326 ( $2.0\times$ )
+ Stage-2 ( $K$ )	$32$ ms ( $3.7\times$ )	$22\%$	1024 ∗	2993 ( $4.5\times$ )
Qwen3-Omni-30B (40-min), aggressive (MCQ) $.85/.35$
full ( $N$ )	$3.6$ s	$100\%$	8	$\sim$ 28

Appendix

Table I.2: Which stage produces the gain depends on the model. We compare Stage 1 alone (shortlist

K_{1}

, no layer-2 cut) with the full pipeline (cut to

K

) on one GH200, on

5

minutes of audio for Qwen 3B and

40

minutes for Qwen 30B. Each ratio is against the full audio of the same run, and run-to-run noise on Qwen 30B is

\pm 1

tok/s. Bold marks the full pipeline’s serving figures, except the Qwen 3B stream count, which reaches the search ceiling ( ∗ ).

Benchmark	Model	audio full	ours (conserv.)	ours (aggr.)	Whisper $\to$ text
MMAU-mini sound	Qwen 3B	.62	.62	.59	.44
	Qwen 30B	.66	.63	.66	.51
MMAR sound	Qwen 3B	.54	.54	.48	.39
	Qwen 30B	.66	.66	.62	.42
IEMOCAP emotion	Qwen 3B	.62	.62	.54	.57
	Qwen 30B	.65	.58	.53	.56

Appendix

Table I.3: LALM against an ASR

\to

text cascade. The same Qwen language model answers zero-shot from the audio or from a Whisper-small.en transcript (

n{=}200

;

204

on the lexical control). The benchmarks cover non-speech sound (MMAU-mini, MMAR), acted-speech emotion (IEMOCAP [ 40 ] ) and a lexical control drawn from MMSU, whose answer is in the words. Ours runs at each model’s MMSU points of Tab. H.3 , unchanged on every benchmark here:

.85/.65

and

.35/.25

on Qwen 3B, and

.85/.55

and

.85/.35

on Qwen 30B. Bold marks audio (full or our conservative cut) that is ahead of the cascade. The control is not bolded.

Model	Audio	ours (compressed)	cascade (Whisper $+$ text)	speedup
Qwen 3B	30 s	$48$ ms	$1.07$ s	$22\times$
	1 min	$61$ ms	$2.40$ s	$\mathbf{39\times}$
	5 min	$497$ ms	$13.9$ s	$28\times$
Qwen 30B	30 s	$1.71$ s	$2.62$ s	$1.5\times$
	5 min	$1.80$ s	$16.1$ s	$8.9\times$
	10 min	$1.97$ s	$30.8$ s	$\mathbf{16\times}$

Appendix

Table I.4: Time to first token, our pipeline against the cascade. Both run on the same clip on one GH200, at the aggressive multiple-choice points (

.50/.20

on Qwen 3B,

.85/.35

on Qwen 30B). Ours is the audio encoder plus the compressed prefill, and the cascade is Whisper-small.en transcription plus the same language model on the transcript, so both encoders are counted. Speedup is cascade over ours, not over full audio. Qwen 3B’s margin falls after one minute because its encoder then takes longer than its whole language-model prefill. Bold marks the largest speedup in each block.

Explore similar work

Apr 26, 2026cs.SD

HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models

Recent large audio language models (LALMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet incur high inference costs. Token compression is an effective method that directly reduces redundant tokens in the sequence. Existing compression methods usually assume that all attention heads in LALMs contribute equally to various audio tasks and calculate token importance by averaging scores across all heads. However, our analysis demonstrates that attention heads exhibit distinct behaviors across diverse audio domains. We further reveal that only a sparse subset of attention heads actively responds to audio, with completely different performance when handling semantic and acoustic tasks. In light of this observation, we propose HeadRouter, a head-importance-aware token pruning method that perceives the varying importance of attention heads in different audio tasks to maximize the retention of crucial tokens. HeadRouter is training-free and can be applied to various LALMs. Extensive experiments on the AudioMarathon and MMAU-Pro benchmarks demonstrate that HeadRouter achieves state-of-the-art compression performance, exceeding the baseline model even when retaining 70% of the audio tokens and achieving 101.8% and 103.0% of the vanilla average on Qwen2.5-Omni-3B and Qwen2.5-Omni-7B, respectively.

Peize He, Yaodi Luo, Xiaoqian Liu +7

1EPIC Lab, Shanghai Jiao Tong University · 2DAIL Tech · 3Northeastern University +2

Large Audio Language Models Visual Token Pruning Audio Understanding Data Compression Methods

Ask PDF

May 24, 2026cs.CL

Locality Matters for Training-Free Audio Token Compression in Audio-Language Models

Audio-language models (ALMs) are increasingly used for audio captioning, question answering, and open-ended audio understanding, but their inference cost remains high when audio inputs are represented as long prefix-token sequences. These audio prefixes consume context budget, increase memory usage, and make deployment harder in resource-constrained or latency-sensitive settings. Existing training-free audio-token reduction methods mainly rely on fixed pooling or score-based pruning. Fixed pooling is content-agnostic, while score-based pruning can preserve isolated salient tokens but discard nearby acoustic context. We propose Local Temporal Bipartite Merging (LTBM), a training-free encoder-space compression method that merges similar nearby audio tokens under an explicit temporal window constraint. Beyond introducing LTBM, we use a controlled Global Merge variant to isolate whether temporal locality itself is a useful inductive bias for audio-token compression. Experiments on AudioCaps, Clotho, and MMAU with Qwen2-Audio show evidence of a task-dependent locality effect: locality-aware merging is more favorable for captioning at several compression settings, especially under stronger compression, while global matching is more competitive for multiple-choice audio understanding. A cross-backbone validation on Audio Flamingo 3 further supports the captioning-side advantage of locality-aware merging under moderate and aggressive compression.

Jiale Luo, Xiaoyu Liang, Haoji Hu

1Zhejiang University

Audio Understanding Token Compression Audio Understanding Training-Free

Ask PDF

Aug 3, 2026cs.AI

Allocation Before Ranking: Decoupled Token Compression for OmniLLMs

Token compression in OmniLLMs is typically posed as a single saliency-ranking problem: score each multimodal token, keep the top-K. We argue this abstraction is mis-specified. The same attention score simultaneously decides two things: how much retained capacity each modality receives, and which tokens within a modality are kept. A shared top-K rule therefore inherits this audio-favoring allocation prior, spending retained capacity on audio before video tokens have a chance to compete. We propose Macer, a training-free compressor that first assigns explicit audio and video budgets, then performs allocation-normalized ranking within each modality at modality-specific shallow layers. Macer significantly reduces token cost while preserving accuracy across audio-grounded, audio--video joint, visual-dominant, and video-centric benchmarks. At 25 % retention, Macer preserves 98.7 % of full-token performance on Qwen2.5-Omni-7B and 97.3 % on Qwen2.5-Omni-3B. On Qwen2.5-Omni-7B, this 25 % setting reaches OmniZip-level performance at 45 % retention while using lower FLOPs. On OmniVinci-9B, the same allocation-before-ranking principle improves over shared top-K ranking by up to 12.9 points.

Zhenghui Guo, Yilin Yang, Yuanbin Man +5

Department of Computer Science, University of Houston · Department of Computer Science and Engineering, The University of Texas at Arlington

Token Compression Data Compression Methods Audio Tokenizers Long-Video Benchmarks Multimodal Large Language Models Ranking

Ask PDF