Transformer-based models have shown strong performance across diverse tasks, but their reliance on self-attention mechanisms often leads to high computational costs. Token sparsification, which reduces the number of tokens processed in transformer layers, offers a promising solution to maintain performance while improving efficiency. However, although widely studied in the vision and language domains, token sparsification remains underexplored in audio, with existing work often limited to a single method, model size, or dataset. This work presents a comprehensive evaluation of token sparsification techniques for audio transformers. We assess a range of token pruning and merging strategies across multiple datasets and model sizes, under both post-training and fine-tuning settings. Furthermore, we introduce a novel sparsification approach that combines attention scores with spectral cues derived from the frequency content of token embeddings. Our experiments show that frequency-aware approaches, particularly those based on spectral flatness and low-frequency energy, consistently outperform attention-only methods. We also find that sparsification based on spectrogram patch intensity is especially effective for longer audio inputs. These results highlight the value of incorporating frequency-domain and input-specific information into sparsification strategies to enhance performance in audio tasks.
Figures & tables
Figure 1: Post-training sparsification results on the AST-Base Gong et al. (2021) model, evaluated on AudioSet-20K Gemmeke et al. (2017) at a 50% keep rate. Methods that include frequency-domain cues, such as the proposed Spectral Flatness (SF) and the adapted Low-Frequency Energy (LFE) Wang et al. (2022b) , outperform others in terms of mean average precision (mAP), a standard metric for multi-label classification. The baseline (without sparsification) achieves an average mAP of 0.3572. The Pareto front is highlighted to illustrate the performance-efficiency trade-off.
Figure 2: Intensity-guided token pruning. Tokens are divided into high- ( H ) and low- ( L ) intensity groups based on an applied threshold. The high-intensity set is then ranked using CLS attention scores, and the Top- K tokens are selected. If fewer than K tokens are available in H , the remaining slots are filled from the low-intensity set.
Figure 3: Low-frequency Energy token pruning. Token embeddings are transformed to the frequency domain, where a Gaussian low-pass filter highlights low-frequency structure. The resulting energy scores are fused with attention scores to guide top-K token selection.
Figure 4: Spectral flatness-guided token pruning. Token embeddings undergo Fast Fourier Transform (FFT) along the embedding dimension to compute spectral flatness, capturing tonal richness. These scores are combined with CLS attention to rank token importance.
Figure 5: Spectral flatness at the token level. (a) A spectrogram with two highlighted tokens. (b) FFT of corresponding token embeddings shows Token A has tonal peaks, while Token B has a flatter, noise-like spectrum - making Token A potentially more informative.
Table 1: GFLOPs of AST-B model on AS-20K after sparsification across keep rates ( r ). ↓ denotes percentage reductions over baseline. Maximum reductions per keep rate are highlighted in green .
Table 2: Post-training sparsification results across model sizes.
Figure 6: Post-training sparsification results on the AST-B Gong et al. (2021) model evaluated on SPC and ESC50 at a 50% keep rate. Methods leveraging frequency-domain cues, such as Spectral Flatness (SF) and Low-Frequency Energy (LFE) Wang et al. (2022b) , outperform other methods in terms of accuracy.
Figure 7: Sparsified Fine-tuning: Performance across varying keep rates. Frequency-based methods (SF, LFE), as well as pure attention-based pruning methods (Top-K, EViT), consistently maintain high performance across tasks. Most methods retain competitive performance down to a 50% keep rate. Overall, the performance differences between methods are not very significant, indicating that multiple sparsification methods can be equally effective when fine-tuned.
Figure 8: Performance comparison of frequency-based methods (LFE and SF) against attention-only baselines (TopK-Wt and TopK-CLS). SF slightly but consistently outperforms attention-based baselines across keep rates, while LFE shows competitive results but occasionally underperforms on ESC50 and AS-20K.
Figure 9: Token pruning performance as a function of input duration. Intensity Pruning consistently outperforms Top-K pruning for longer audio windows ( >2 seconds), suggesting that spectrogram intensity is a more informative proxy for longer audio samples.
Audio token compression reduces the number of tokens or bits needed to represent sound while preserving information for listening, generation, and understanding.
Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder's representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top-k token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.
Shreya Sajid, Anna Vettoruzzo, Joaquin Vanschoren
Department of Mathematics & Computer Science Eindhoven University of Technology Eindhoven, Netherlands
This paper proposes SpecAugment-Patch Merging, a simple yet effective method to accelerate Audio Spectrogram Transformer (AST) training. We first apply SpecAugment to mask input spectrograms at the patch level, and after positional embeddings are added, the method selects r pairs of masked patches and merges them, reducing the number of tokens processed by the Transformer. Increasing the number of merged pairs r from 0 to 100 keeps mAP on AudioSet nearly unchanged (34.07 to 34.08) while throughput increases from 43.3 to 49.3 samples/sec, which is a relatively 13.9% improvement. Similar patterns appear on ESC-50 and Speech Commands V2, where throughput steadily improves with only minor accuracy changes, demonstrating that this merging approach provides faster training with minimal performance loss.
Recent large audio language models (LALMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet incur high inference costs. Token compression is an effective method that directly reduces redundant tokens in the sequence. Existing compression methods usually assume that all attention heads in LALMs contribute equally to various audio tasks and calculate token importance by averaging scores across all heads. However, our analysis demonstrates that attention heads exhibit distinct behaviors across diverse audio domains. We further reveal that only a sparse subset of attention heads actively responds to audio, with completely different performance when handling semantic and acoustic tasks. In light of this observation, we propose HeadRouter, a head-importance-aware token pruning method that perceives the varying importance of attention heads in different audio tasks to maximize the retention of crucial tokens. HeadRouter is training-free and can be applied to various LALMs. Extensive experiments on the AudioMarathon and MMAU-Pro benchmarks demonstrate that HeadRouter achieves state-of-the-art compression performance, exceeding the baseline model even when retaining 70% of the audio tokens and achieving 101.8% and 103.0% of the vanilla average on Qwen2.5-Omni-3B and Qwen2.5-Omni-7B, respectively.
Peize He, Yaodi Luo, Xiaoqian Liu +7
1EPIC Lab, Shanghai Jiao Tong University · 2DAIL Tech · 3Northeastern University +2