Muting Tokens: Exploring Token Sparsification for Efficient Audio Transformers

Shreya Sajid; Fabrizio Julian Piva; Hiram Rayo Torres Rodriguez; Elif Ceren Gok Yildirim; Murat Onur Yildirim; Joaquin Vanschoren

cs.SDOct 22, 2026

Muting Tokens: Exploring Token Sparsification for Efficient Audio Transformers

Authors: Shreya Sajid, Fabrizio Julian Piva, Hiram Rayo Torres Rodriguez, Elif Ceren Gok Yildirim, Murat Onur Yildirim, Joaquin Vanschoren

Organizations: Eindhoven University of Technology, Eindhoven, Netherlands · NXP Semiconductors, Eindhoven, Netherlands

Abstract

Transformer-based models have shown strong performance across diverse tasks, but their reliance on self-attention mechanisms often leads to high computational costs. Token sparsification, which reduces the number of tokens processed in transformer layers, offers a promising solution to maintain performance while improving efficiency. However, although widely studied in the vision and language domains, token sparsification remains underexplored in audio, with existing work often limited to a single method, model size, or dataset. This work presents a comprehensive evaluation of token sparsification techniques for audio transformers. We assess a range of token pruning and merging strategies across multiple datasets and model sizes, under both post-training and fine-tuning settings. Furthermore, we introduce a novel sparsification approach that combines attention scores with spectral cues derived from the frequency content of token embeddings. Our experiments show that frequency-aware approaches, particularly those based on spectral flatness and low-frequency energy, consistently outperform attention-only methods. We also find that sparsification based on spectrogram patch intensity is especially effective for longer audio inputs. These results highlight the value of incorporating frequency-domain and input-specific information into sparsification strategies to enhance performance in audio tasks.

Figures & tables

Figure 1: Post-training sparsification results on the AST-Base Gong et al. (2021) model, evaluated on AudioSet-20K Gemmeke et al. (2017) at a 50% keep rate. Methods that include frequency-domain cues, such as the proposed Spectral Flatness (SF) and the adapted Low-Frequency Energy (LFE) Wang et al. (2022b) , outperform others in terms of mean average precision (mAP), a standard metric for multi-label classification. The baseline (without sparsification) achieves an average mAP of 0.3572. The Pareto front is highlighted to illustrate the performance-efficiency trade-off.

Figure 2: Intensity-guided token pruning. Tokens are divided into high- (

H

) and low- (

L

) intensity groups based on an applied threshold. The high-intensity set is then ranked using CLS attention scores, and the Top-

K

tokens are selected. If fewer than

K

tokens are available in

H

, the remaining slots are filled from the low-intensity set.

Figure 3: Low-frequency Energy token pruning. Token embeddings are transformed to the frequency domain, where a Gaussian low-pass filter highlights low-frequency structure. The resulting energy scores are fused with attention scores to guide top-K token selection.

Figure 4: Spectral flatness-guided token pruning. Token embeddings undergo Fast Fourier Transform (FFT) along the embedding dimension to compute spectral flatness, capturing tonal richness. These scores are combined with CLS attention to rank token importance.

Panel of Figure 5 — Figure 5: Spectral flatness at the token level. (a) A spectrogram with two highlighted tokens. (b) FFT of corresponding token embeddings shows Token A has tonal peaks, while Token B has a flatter, noise-like spectrum - making Token A potentially more informative.

	AS-20K (GFLOPs)
Baseline	261.43
r (%)	90	70	50	25
Top-K Haurum et al. (2022a)	$222.28^{{\color[rgb]{0,0.5,0}\text{$ \downarrow $15.0\%}}}$	$164.16^{{\color[rgb]{0,0.5,0}\text{$ \downarrow $37.2\%}}}$	$124.85^{{\color[rgb]{0,0.5,0}\text{$ \downarrow $52.3\%}}}$	$93.74^{{\color[rgb]{0,0.5,0}\text{$ \downarrow $64.3\%}}}$
EViT Liang et al. (2022a)	$222.61^{\text{$ \downarrow $14.8\%}}$	$164.45^{\text{$ \downarrow $37.1\%}}$	$125.05^{\text{$ \downarrow $52.2\%}}$	$93.99^{\text{$ \downarrow $64.0\%}}$
ToMe Bolya et al. (2023)	$226.11^{\text{$ \downarrow $13.5\%}}$	$165.77^{\text{$ \downarrow $36.6\%}}$	$125.60^{\text{$ \downarrow $52.0\%}}$	–
Intensity	$222.28^{{\color[rgb]{0,0.5,0}\text{$ \downarrow $15.0\%}}}$	$164.16^{{\color[rgb]{0,0.5,0}\text{$ \downarrow $37.2\%}}}$	$124.89^{\text{$ \downarrow $52.2\%}}$	$93.79^{\text{$ \downarrow $64.1\%}}$

Table 1: GFLOPs of AST-B model on AS-20K after sparsification across keep rates ( r ).

\downarrow

denotes percentage reductions over baseline. Maximum reductions per keep rate are highlighted in green .

Table 2: Post-training sparsification results across model sizes.

Figure 6: Post-training sparsification results on the AST-B Gong et al. (2021) model evaluated on SPC and ESC50 at a 50% keep rate. Methods leveraging frequency-domain cues, such as Spectral Flatness (SF) and Low-Frequency Energy (LFE) Wang et al. (2022b) , outperform other methods in terms of accuracy.

Panel of Figure 7 — Figure 7: Sparsified Fine-tuning: Performance across varying keep rates. Frequency-based methods (SF, LFE), as well as pure attention-based pruning methods (Top-K, EViT), consistently maintain high performance across tasks. Most methods retain competitive performance down to a 50% keep rate. Overall, the performance differences between methods are not very significant, indicating that multiple sparsification methods can be equally effective when fine-tuned.

Panel of Figure 8 — Figure 8: Performance comparison of frequency-based methods (LFE and SF) against attention-only baselines (TopK-Wt and TopK-CLS). SF slightly but consistently outperforms attention-based baselines across keep rates, while LFE shows competitive results but occasionally underperforms on ESC50 and AS-20K.

Panel of Figure 9 — Figure 9: Token pruning performance as a function of input duration. Intensity Pruning consistently outperforms Top-K pruning for longer audio windows ( $>2$ seconds), suggesting that spectrogram intensity is a more informative proxy for longer audio samples.

Audio Token Compression

Audio token compression reduces the number of tokens or bits needed to represent sound while preserving information for listening, generation, and understanding.

See all 37 papers

AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs
Oct 9, 2026Shreya Sajid, Anna Vettoruzzo, Joaquin VanschorenAudio Token Compression Audio-Language Models
Department of Mathematics & Computer Science Eindhoven University of Technology Eindhoven, Netherlands
From Masking to Merging: Rethinking SpecAugment for Efficient Audio Spectrogram Transformer
Sep 6, 2026Minhee Park, Hyowon Ahn, Chanwoo KimToken Merging Data Augmentation
Korea University, Republic of Korea
HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
Apr 26, 2026Peize He, Yaodi Luo, Xiaoqian Liu +7Audio Token Compression Self-Attention
1EPIC Lab, Shanghai Jiao Tong University · 2DAIL Tech · 3Northeastern University +2

Oct 9, 2026J/K move · Enter open · S save

AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

Shreya Sajid, Anna Vettoruzzo, Joaquin Vanschoren

Department of Mathematics & Computer Science Eindhoven University of Technology Eindhoven, Netherlands

Audio Token Compression Audio-Language Models Token Pruning LLM Inference Acceleration

Open paper AskPDF arXiv

Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder's representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top-

k

token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.

Abstract

Audio Token Compression

Explore similar work

AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

From Masking to Merging: Rethinking SpecAugment for Efficient Audio Spectrogram Transformer

HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models