AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

Shreya Sajid; Anna Vettoruzzo; Joaquin Vanschoren

cs.SDOct 9, 2026

AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

Authors: Shreya Sajid, Anna Vettoruzzo, Joaquin Vanschoren

Organizations: Department of Mathematics & Computer Science Eindhoven University of Technology Eindhoven, Netherlands

Abstract

Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder's representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top- $k$ token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.

Figures & tables

Figure 1: Overview of AudioTETRIS. A phoneme-transition-aware acoustic prior $w_{t}$ is computed from the audio encoder’s hidden states via the transitional distance $d_{t}=\|h_{t}-h_{t-1}\|_{2}^{2}$ , preserving phoneme transitions $y_{\text{hard}}$ (§ 4 ; left). Token pruning occurs in two stages: Stage 1 injects $w_{t}$ as a static acoustic prior from the encoder’s hidden states; Stage 2 computes a depth-adaptive LLM attention score, fusing intra-audio attention in early layers with cross-modal (text-to-audio) attention in later layers (right). The two scores are fused into a single importance score, and top- $k$ selection retains the most salient tokens at each pruning layer $\ell$ .

Figure 2: Distribution of decoder attention mass allocated to audio tokens across model depth in Qwen2-Audio on LibriSpeech. Despite audio tokens comprising

84.98\%

of the input sequence (dashed ring), early layers are dominated by intra-audio attention (

76.91\%

), which drops sharply in middle (

46.73\%

) and late layers (

40.60\%

) as attention shifts to text-conditioned representations. This shift motivates our layer-dependent pooling rule in Eq. 3 .

Figure 3: Layer-wise diagnostics across Qwen2-Audio, Phi-4-Multimodal, and Audio Flamingo 3. Top: Mean correlation between

b_{t}

and ground-truth phoneme transitions (

r_{\text{soft}}

and

r_{\text{hard}}

). Bottom: Kurtosis (

\kappa

) of the distance distribution

d_{t}

. Both metrics identify an intrinsic mid-network sweet spot (shaded) where phonetic transitions are maximally accentuated.

Method	LibriSpeech WER $\downarrow$	IEMOCAP Acc. $\uparrow$	MMAU-mini Acc. $\uparrow$	CoVoST2 BLEU $\uparrow$	Latency (s)	TFLOPs
No Compression	$4.61^{\pm 0.12}$	$55.11^{\pm 1.06}$	$42.20^{\pm 0.96}$	$33.03^{\pm 0.72}$	$1.31$	$7.89$
Compression Ratio 0.36
Random	$9.91^{\pm 1.67}$	$53.10^{\pm 0.37}$	$40.18^{\pm 0.23}$	$29.10^{\pm 0.02}$	$1.15$	$6.93$
SpeechPrune	$6.75^{\pm 1.44}$	$56.43^{\pm 0.35}$	$41.17^{\pm 0.60}$	$31.34^{\pm 0.07}$	$1.21$	$6.99$
SparseVLM †‡	$8.40^{\pm 1.08}$	$52.37^{\pm 0.21}$	$41.09^{\pm 0.73}$	$31.38^{\pm 0.12}$	$1.16$	$7.28$
LTBM	$5.90^{\pm 0.48}$	$55.90^{\pm 0.27}$	$40.67^{\pm 0.67}$	$29.97^{\pm 0.06}$	$1.25$	$6.93$

Table 1: Performance and efficiency comparison across compression ratios on Qwen2Audio evaluated on LibriSpeech (ASR), IEMOCAP (Emotion Recognition), MMAU-mini (Audio Understanding, MCQ), and CoVoST2 (Speech Translation). WER (%) is lower-is-better (

\downarrow

); Accuracy (%) and BLEU are higher-is-better (

\uparrow

). Baselines: SpeechPrune ( Lin et al., 2025 ) , SparseVLM ( Zhang et al., 2025b ) , LTBM ( Luo et al., 2026 ) .

Figure 4: Comparison of LibriSpeech WER (

\downarrow

) and MMAU-mini accuracy (

\uparrow

) at compression ratio 0.51, for Phi-4-Multimodal and Audio Flamingo 3. Error bars show standard deviation across runs. AudioTETRIS is highlighted in the darker shade.

Figure 5: WER/accuracy/BLEU across distance metrics for the transition score, at compression ratio

0.51

. Error bars show standard deviation over runs.

Figure 6: Ablation of AudioTETRIS stages at compression ratio

0.51

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Checkpoint	License
Qwen2-Audio ( Chu et al., 2024 )	Qwen/Qwen2-Audio-7B-Instruct	Apache 2.0
Phi-4-Multimodal ( Microsoft et al., 2025 )	microsoft/Phi-4-multimodal-instruct	MIT
Audio Flamingo 3 ( Ghosh et al., 2025a )	nvidia/audio-flamingo-3-hf	NVIDIA OneWay Noncommercial License *

Appendix

Table 2: Pretrained checkpoints used in this work.

Dataset	Split used	Original license
LibriSpeech (test-other) ( Panayotov et al., 2015 )	AudioLLMs/librispeech_test_other , test	CC BY 4.0
IEMOCAP ( Busso et al., 2008 )	AudioLLMs/iemocap_emotion_recognition , test	Custom non-commercial research license (USC SAIL)
MMAU-mini ( Sakshi et al., 2025 )	AudioLLMs/MMAU-mini , test	Not explicitly stated by source
CoVoST2 (En $\to$ Zh) ( Wang et al., 2021 )	AudioLLMs/covost2_en_zh_test , test	CC0 (per original release); Hub mirror tagged CC BY-NC 4.0

Appendix

Table 3: Evaluation datasets, splits, and licensing.

Figure 7: Per-layer intra-audio (

a_{\text{intra}}(\ell)

) and cross-modal (

a_{\text{cross}}(\ell)

) attention mass on the LibriSpeech test-other calibration subset, for Qwen2-Audio ( Chu et al., 2024 ) , Phi-4-Multimodal ( Microsoft et al., 2025 ) , and Audio Flamingo 3 ( Ghosh et al., 2025a ) The dashed vertical line marks the identified crossover layer

\ell^{\dagger}

for each backbone. Intra-audio attention dominates in early layers and is progressively overtaken by cross-modal attention as depth increases, consistent with Figure 2 .

Figure 8: Schematic of the phoneme-transition-aware prior pipeline. Top: audio waveform with forced-alignment phoneme segments. Second row: per-token encoder hidden states

h_{t}

, colored by phoneme membership. Third row: raw derivative

d_{t}

(bars) and the Gaussian-smoothed transition score

b_{t}

(line). Bottom: ground-truth transition labels, shown as discrete spikes

y_{\text{hard}}

and their Gaussian-smoothed form

y_{\text{soft}}

, used to compute

r_{\text{hard}}

and

r_{\text{soft}}

Figure 9: Layer-wise boundary correlation (

r_{\text{soft}}

r_{\text{hard}}

) across Qwen2-Audio, Phi-4-Multimodal, and Audio Flamingo 3, computed on CoVoST2 ( top , MFA alignments generated in-house) and IEMOCAP ( bottom , official released alignments). Shaded regions mark each backbone’s mid-network sweet spot, consistent with Figure 3 . Dashed vertical lines and annotated values mark the peak layer for each curve.

Figure 10: Sensitivity of the transition-prior correlation to the Gaussian smoothing width

\sigma

, on all backbones. Each panel sweeps

\sigma

(encoder tokens, with milliseconds on the secondary top axis) and reports

r_{\text{soft}}

and

r_{\text{hard}}

at that backbone’s mid-network sweet-spot layer. The shaded band and dashed line mark

\sigma=2

, the value used throughout the main text.

Figure 11: Qualitative analysis of Qwen2-Audio encoder token embedding trajectories for a single LibriSpeech utterance, projected via t-SNE and colored by phoneme identity, at early, middle, and late encoder depths. Colored points denote individual tokens where different colors represent different phonemes; Star markers indicate frames at phoneme transitions per forced alignment; Faint gray points show other utterances’ frames for spatial context.

Method	LibriSpeech WER $\downarrow$	IEMOCAP Acc. $\uparrow$	MMAU-mini Acc. $\uparrow$	CoVoST2 BLEU $\uparrow$	Latency (s)	TFLOPs
No Compression	$4.85^{\pm 0.34}$	$39.34^{\pm 0.29}$	$54.0^{\pm 0.85}$	$36.06^{\pm 0.07}$	$1.15$	$1.403$
Compression Ratio 0.36
Random	$5.64^{\pm 1.92}$	$39.00^{\pm 0.51}$	$52.60^{\pm 0.89}$	$19.05^{\pm 0.05}$	$0.97$	$0.96$
SpeechPrune	$5.55^{\pm 0.06}$	$38.28^{\pm 1.10}$	$53.70^{\pm 0.44}$	$22.60^{\pm 0.07}$	$1.01$	$0.96$
SparseVLM †‡	$5.43^{\pm 0.79}$	$39.75^{\pm 0.36}$	$52.70^{\pm 0.43}$	$22.05^{\pm 0.07}$	$1.02$	$1.00$
LTBM	$6.60^{\pm 0.00}$	$40.40^{\pm 0.84}$	$54.40^{\pm 0.00}$	$26.76^{\pm 0.19}$	$1.00$	$0.96$

Appendix

Table 4: Extended results for Phi-4-Multimodal across all compression ratios.

Method	LibriSpeech WER $\downarrow$	IEMOCAP Acc. $\uparrow$	MMAU-mini Acc. $\uparrow$	CoVoST2 BLEU $\uparrow$	Latency (s)	TFLOPs
No Compression	$3.61^{\pm 0.54}$	$59.11^{\pm 0.87}$	$62.09^{\pm 0.45}$	$37.02^{\pm 0.71}$	$1.29$	$7.93$
Compression Ratio 0.36
Random	$4.02^{\pm 1.21}$	$58.43^{\pm 0.63}$	$60.97^{\pm 0.23}$	$36.82^{\pm 0.10}$	$1.13$	$6.92$
SpeechPrune	$4.01^{\pm 2.10}$	$58.23^{\pm 0.59}$	$61.42^{\pm 0.83}$	$36.73^{\pm 0.05}$	$1.15$	$7.09$
SparseVLM †‡	$3.78^{\pm 1.45}$	$57.34^{\pm 0.32}$	$58.83^{\pm 0.68}$	$36.18^{\pm 0.06}$	$1.14$	$7.31$
LTBM	$3.83^{\pm 0.38}$	$58.11^{\pm 0.41}$	$61.00^{\pm 0.65}$	$35.89^{\pm 0.02}$	$1.21$	$6.98$

Appendix

Table 5: Extended results for Audio Flamingo 3 across all compression ratios.

Method	LibriSpeech WER $\downarrow$	IEMOCAP Acc. $\uparrow$	MMAU-mini Acc. $\uparrow$	CoVoST2 BLEU $\uparrow$
Compression Ratio 0.36
Only Stage 1	$5.50^{\pm 0.25}$	$57.45^{\pm 0.39}$	$41.80^{\pm 0.62}$	$31.21^{\pm 0.09}$
Only Stage 2	$9.21^{\pm 0.36}$	$57.30^{\pm 1.19}$	$41.70^{\pm 0.80}$	$30.93^{\pm 0.04}$
Stage 1 + Stage 2	$\mathbf{4.52^{\pm 0.21}}$	$\mathbf{58.04^{\pm 0.84}}$	$\mathbf{42.36^{\pm 0.21}}$	$\mathbf{31.96^{\pm 0.04}}$
Compression Ratio 0.51
Only Stage 1	$7.39^{\pm 0.09}$	$56.37^{\pm 1.55}$	$41.90^{\pm 0.42}$	$31.18^{\pm 0.08}$

Appendix

Table 6: Ablation results comparing individual stages against full AudioTETRIS across different compression ratios. The best-performing combination per compression ratio block is highlighted.

Schedule	Layers	$n$ stages	Split
Single-stage @ 1	1	1	–
Single-stage @ 15	15	1	–
Two-stage $\{1,15\}$ (default)	1, 15	2	Uniform
Two-stage $\{1,19\}$	1, 19	2	Uniform
Three-stage $\{1,8,15\}$	1, 8, 15	3	Uniform
Front-loaded	1, 15	2	$w=(0.7,0.3)$

Appendix

Table 7: Candidate pruning schedules. Placement variants use the uniform split; front-/back-loaded vary only

w_{i}

at the default layers

\{1,15\}

Schedule	Layers	rate 0.36	rate 0.51	rate 0.64	rate 0.75
Single-stage @ 1	1	36%	51%	64%	75%
Single-stage @ 15	15	36%	51%	64%	75%
Two-stage (default)	1, 15	20% ( $\times$ 2)	30% ( $\times$ 2)	40% ( $\times$ 2)	50% ( $\times$ 2)
Two-stage $\{1,19\}$	1, 19	20% ( $\times$ 2)	30% ( $\times$ 2)	40% ( $\times$ 2)	50% ( $\times$ 2)
Three-stage	1, 8, 15	14% ( $\times$ 3)	21% ( $\times$ 3)	29% ( $\times$ 3)	37% ( $\times$ 3)
Front-loaded	1, 15	27%/13%	39%/19%	51%/26%	62%/34%

Appendix

Table 8: Per-stage token-drop rate (% of currently live tokens removed at that stage) for each schedule, at every evaluated compression rate. Equal per-stage rates are written once as “

X\%

(

\times n

)”; front-/back-loaded list rates stage by stage.

Schedule	rate 0.36	rate 0.51	rate 0.64	rate 0.75
Single-stage @ 1	$6.51^{\pm 0.14}$	$8.92^{\pm 0.17}$	$12.40^{\pm 0.36}$	$18.78^{\pm 0.41}$
Single-stage @ 15	$5.44^{\pm 0.61}$	$6.93^{\pm 0.42}$	$8.94^{\pm 0.48}$	$12.89^{\pm 0.82}$
Two-stage $\{1,15\}$ (default)	$4.52^{\pm 0.21}$	$\mathbf{5.77^{\pm 0.33}}$	$6.62^{\pm 0.29}$	$8.50^{\pm 0.18}$
Two-stage $\{1,19\}$	$4.78^{\pm 0.38}$	$6.02^{\pm 0.35}$	$6.94^{\pm 0.62}$	$8.96^{\pm 0.31}$
Three-stage $\{1,8,15\}$	$\mathbf{4.36^{\pm 0.71}}$	$5.93^{\pm 0.71}$	$\mathbf{6.47^{\pm 0.72}}$	$\mathbf{8.12^{\pm 0.73}}$
Front-loaded	$4.99^{\pm 0.69}$	$6.72^{\pm 0.76}$	$7.68^{\pm 0.29}$	$10.73^{\pm 0.28}$

Appendix

Table 9: Schedule ablation, LibriSpeech WER (%, lower is better), Qwen2-Audio, across compression rates.

Schedule	WER	IEMOCAP	MMAU	CoVoST	Latency	TFLOPs
Single-stage @ 1	$8.92^{\pm 0.17}$	$53.55^{\pm 0.50}$	$39.81^{\pm 0.41}$	$27.05^{\pm 0.05}$	$\mathbf{0.98}$	$\mathbf{5.51}$
Single-stage @ 15	$6.93^{\pm 0.42}$	$55.83^{\pm 0.52}$	$41.52^{\pm 0.24}$	$29.53^{\pm 0.09}$	$0.99$	$5.92$
Two-stage $\{1,15\}$ (default)	$\mathbf{5.77^{\pm 0.33}}$	$57.03^{\pm 0.41}$	$42.91^{\pm 0.36}$	$31.45^{\pm 0.06}$	$1.00$	$6.13$
Two-stage $\{1,19\}$	$6.02^{\pm 0.35}$	$57.41^{\pm 0.47}$	$43.18^{\pm 0.18}$	$\mathbf{31.69^{\pm 0.07}}$	$1.03$	$6.13$
Three-stage $\{1,8,15\}$	$5.93^{\pm 0.71}$	$55.68^{\pm 0.37}$	$41.67^{\pm 0.72}$	$31.45^{\pm 0.10}$	$1.04$	$6.32$
Front-loaded	$6.72^{\pm 0.76}$	$56.91^{\pm 0.19}$	$41.48^{\pm 0.39}$	$30.20^{\pm 0.18}$	$1.02$	$5.71$

Appendix

Table 10: Efficiency and downstream-task trade-offs at compression rate 0.51. Latency is relative to the uncompressed baseline (

1.00\times

Panel of Figure 12 — Figure 12: Token survival at the final pruning layer for three representative utterances (ground-truth transcripts shown under each panel). Rows within each panel (top to bottom): mel spectrogram, Only Stage 1 (phoneme-transitions only), Only Stage 2 (attention-only), Stage 1 + Stage 2 (attention $\times$ phoneme-transitions, our full method). Orange = kept uniquely by Only Stage 1; mauve = kept uniquely by Only Stage 2; blue = kept uniquely by Stage 1 + Stage 2; grey = pruned.

Audio Token Compression

Audio token compression reduces the number of tokens or bits needed to represent sound while preserving information for listening, generation, and understanding.

See all 35 papers

Sep 30, 2026

Audio Token Attention Is Predictable Before the Language Model Runs

A large audio language model (LALM) turns a minute of speech into 750-1,500 tokens and prefills every one. Image-token pruning often cuts after the language model's first layers, where image tokens draw little attention. Audio tokens draw much more attention there, and their ranking is still far from final, so audio needs a ranking before the language model runs. Surprisingly, the attention an audio token will receive across the language model is already linearly predictable from its encoder output, before the language model runs. A linear map, fitted in closed form without labels, predicts this all-layer attention ranking at

ρ\geq .69

on eleven of thirteen LALMs. Our method, Triage, cuts audio tokens by this prediction and, on multiple choice, cuts again at layer 2, correcting the prediction with the attention observed there. Triage sets its compression without labels, under two budgets that limit how far its output may differ from the model's own full-audio output. At the conservative budget, its word error rate and accuracy stay within .04 of full audio. At the aggressive budget, Triage beats every baseline in all twelve transcription cases. On multiple choice, at 2.2-5x compression, it outperforms DART, the strongest baseline on average, by .043 in mean accuracy. Because it cuts before the language model, it raises the audio that fits in Qwen2.5-Omni-3B's context window from 21.8 to about 62 minutes. At its most compressive point, Triage lets one GPU serve 4x as many concurrent 5-minute streams of that model. Project page: https://audio-triage.github.io

Kyoungjun Park, Yunzhe Li, Lili Qiu

The University of Texas at Austin

Audio Token Compression Audio Understanding Audio-Language Models Token Pruning LLM Inference Acceleration Token Compression

Ask PDF

May 24, 2026

Locality Matters for Training-Free Audio Token Compression in Audio-Language Models

Audio-language models (ALMs) are increasingly used for audio captioning, question answering, and open-ended audio understanding, but their inference cost remains high when audio inputs are represented as long prefix-token sequences. These audio prefixes consume context budget, increase memory usage, and make deployment harder in resource-constrained or latency-sensitive settings. Existing training-free audio-token reduction methods mainly rely on fixed pooling or score-based pruning. Fixed pooling is content-agnostic, while score-based pruning can preserve isolated salient tokens but discard nearby acoustic context. We propose Local Temporal Bipartite Merging (LTBM), a training-free encoder-space compression method that merges similar nearby audio tokens under an explicit temporal window constraint. Beyond introducing LTBM, we use a controlled Global Merge variant to isolate whether temporal locality itself is a useful inductive bias for audio-token compression. Experiments on AudioCaps, Clotho, and MMAU with Qwen2-Audio show evidence of a task-dependent locality effect: locality-aware merging is more favorable for captioning at several compression settings, especially under stronger compression, while global matching is more competitive for multiple-choice audio understanding. A cross-backbone validation on Audio Flamingo 3 further supports the captioning-side advantage of locality-aware merging under moderate and aggressive compression.

Jiale Luo, Xiaoyu Liang, Haoji Hu

1Zhejiang University

Token Merging Audio Token Compression Audio Understanding Audio-Language Models Audio Captioning Multimodal Token Compression

Ask PDF

Apr 26, 2026

HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models

Recent large audio language models (LALMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet incur high inference costs. Token compression is an effective method that directly reduces redundant tokens in the sequence. Existing compression methods usually assume that all attention heads in LALMs contribute equally to various audio tasks and calculate token importance by averaging scores across all heads. However, our analysis demonstrates that attention heads exhibit distinct behaviors across diverse audio domains. We further reveal that only a sparse subset of attention heads actively responds to audio, with completely different performance when handling semantic and acoustic tasks. In light of this observation, we propose HeadRouter, a head-importance-aware token pruning method that perceives the varying importance of attention heads in different audio tasks to maximize the retention of crucial tokens. HeadRouter is training-free and can be applied to various LALMs. Extensive experiments on the AudioMarathon and MMAU-Pro benchmarks demonstrate that HeadRouter achieves state-of-the-art compression performance, exceeding the baseline model even when retaining 70% of the audio tokens and achieving 101.8% and 103.0% of the vanilla average on Qwen2.5-Omni-3B and Qwen2.5-Omni-7B, respectively.

Peize He, Yaodi Luo, Xiaoqian Liu +7

1EPIC Lab, Shanghai Jiao Tong University · 2DAIL Tech · 3Northeastern University +2

Audio Token Compression Self-Attention Audio Understanding Audio-Language Models Token Pruning Multimodal Token Compression+3

Ask PDF

Abstract

Audio Token Compression

Explore similar work

Audio Token Attention Is Predictable Before the Language Model Runs

Locality Matters for Training-Free Audio Token Compression in Audio-Language Models

HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models