Audio Token Compression

Follow this topic Ask about this topic

Audio token compression studies how to represent speech, music, and other sounds with fewer tokens or bits while retaining information needed for tasks such as speech recognition, audio question answering, and reconstruction. Approaches include learned codecs and quantizers, distillation and adaptive segmentation, and training-free token merging or pruning based on attention, temporal structure, or semantic cues; some methods also compress long-context KV caches. Evaluations span ASR sets such as LibriSpeech, audio-understanding benchmarks such as AudioCaps, Clotho, and MMAU, and codec measures of fidelity, intelligibility, and speaker similarity. Large reductions can often preserve much of a model’s performance, but the best compression depends on the task and model, and aggressive reduction can discard details such as emotion, timing, or acoustic fidelity.

Latest papers

37 papers11 papers in the last four weeks, up 83% on the four weeks before. 0.1% of all new papers.

Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models
Apr 8, 2026Bajian Xiang, Tingwei Guo, Xuan Chen +1Speech Language Models Audio Token Compression
Beike Inc., Beijing, China
Abstract Compression of Conversational Audio Context for LLM-based ASR
Mar 27, 2026Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso +7Contextual ASR Automatic Speech Recognition
Idiap Research Institute, Switzerland · EPFL, Switzerland · Uniphore, U.S.A. +2
Emotion-Aware Quantization for Discrete Speech Representations: An Analysis of Emotion Preservation
Mar 22, 2026Haoguang Zhou, Siyi Wang, Jingyao Wu +2Discrete Speech Representation Learning Residual VQ
The University of Melbourne, Australia · Massachusetts Institute of Technology, USA · Monash University, Australia
Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio
Mar 9, 2026Phillip Long, Zachary Novack, Chris DonahueLossless Data Compression Byte-Level Language Model
University of California, San Diego, Computer Science and Engineering Department · Carnegie Mellon University, School of Computer Science
A Generative-First Neural Audio Autoencoder
Feb 17, 2026Jonah Casebeer, Ge Zhu, Zhepei Wang +1Autoencoders Audio Representation Learning
Adobe Research
S-PRESSO: Ultra Low Bitrate Sound Effect Compression With Diffusion Autoencoders And Offline Quantization
Feb 16, 2026Zineb Lahrichi, Gaëtan Hadjeres, Gaël Richard +1Neural Audio Codecs Audio Token Compression
IP Paris, SonyAI · Sony AI · LTCI, Télécom Paris, Institut Polytechnique de Paris +2
Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy
Sep 22, 2025Geewook Kim, Minjoon SeoVLM Evaluation Audio-Visual Understanding
NAVER Cloud AI, South Korea · KAIST AI, South Korea

Apr 8, 2026J/K move · Enter open · S save

Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models

Bajian Xiang, Tingwei Guo, Xuan Chen, Yang Han

Beike Inc., Beijing, China

Speech Language Models Audio Token Compression LLM Inference Acceleration Token Merging

Open paper AskPDF arXiv

Large Speech Language Models (LSLMs) typically operate at high token rates (tokens/s) to ensure acoustic fidelity, yet this results in sequence lengths that far exceed the underlying semantic content, incurring prohibitive inference costs. In this paper, we empirically revisit the necessity of such granular token-level processing. Through layer-wise oracle interventions, we unveil a structured redundancy hierarchy: while shallow layers encode essential acoustic details, deep layers exhibit extreme redundancy, allowing for aggressive compression. Motivated by these findings, we introduce Affinity Pooling, a training-free, similarity-based token merging mechanism. By strategically applying this method at both input and deep layers, we effectively compress speech representations without compromising semantic information. Extensive evaluations across three tasks demonstrate that our approach reduces prefilling FLOPs by 27.48% while maintaining competitive accuracy. Practical deployment further confirms significant efficiency gains, yielding up to

\sim

1.7

\times

memory savings and

\sim

1.1

\times

faster time-to-first-token on long utterances. Our results challenge the necessity of fully distinct token representations, providing new perspectives on LSLM efficiency.