Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio

Phillip Long; Zachary Novack; Chris Donahue

cs.SDMar 9, 2026

Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio

Authors: Phillip Long, Zachary Novack, Chris Donahue

Organizations: University of California, San Diego, Computer Science and Engineering Department · Carnegie Mellon University, School of Computer Science

Abstract

Autoregressive "language" models (LMs) trained on raw waveforms can be repurposed for lossless audio compression, but prior work is limited to 8-bit audio, leaving open whether such approaches work for practical settings (16/24-bit) and can compete with existing codecs. We benchmark LM-based compression on full-fidelity audio across diverse domains (music, speech, bioacoustics), sampling rates (16kHz-48kHz), and bit depths (8, 16, 24-bit). Standard sample-level tokenization becomes intractable at higher bit depths due to vocabulary size (65K for 16-bit; 16.7M for 24-bit). We propose Trilobyte, a byte-level tokenization schema for full resolution audio, improving vocabulary scaling from $O(2^{b})$ to $O(1)$ and enabling the first tractable 24-bit LM-based lossless compression. While LMs consistently outperform FLAC and yield state-of-the-art compression at 8-bit and 16-bit, we observe that compression gains become more modest as bit depth increases beyond 8-bit.

Figures & tables

Figure 1: Tokenization strategies for language model compression. Standard sample-level tokenization (top) yields vocabulary size $|\mathcal{V}|=2^{b}$ . This exponential scaling inhibits modeling of industry-standard bit depths ( $16$ , $24$ ). Trilobyte’s hierarchical byte-level tokenization (bottom) decomposes samples into bytes, yielding constant $|\mathcal{V}|=256$ regardless of bit depth (at the cost of increasing sequence length by $\left\lceil{b/8}\right\rceil$ ). Both feed into an AR LM and arithmetic coder, but Trilobyte enables tractable 24-bit modeling.

$b$ (Bits)	$f_{s}$ (Hz) / $c$ (#Ch)	Domain	Dataset	FLAC	In-context	Standard	Trilobyte	Transfer
8	16000 / 1	Speech	SC09	0.95	1.80	2.08	2.08	2.88
	16000 / 1	Music	Beethoven	1.69	1.33	7.94	7.94	7.45
	16000 / 1		YouTube Mix	1.58	1.28	4.15	4.15	5.14
16	48000 / 1	Speech	VCTK	2.32	1.75	2.66	2.66	2.68
	22050 / 1		LJSpeech	1.69	1.49	1.98	2.08	2.04
	16000 / 1		LibriSpeech	1.74	1.64	2.06	2.11	2.10

Table 1: Compression rates across methods and bit depths. FLAC uses compression level 8. In-context uses pretrained Llama-2-7B [ 15 ] . Standard refers to sample-level tokenization; Trilobyte uses hierarchical byte-level tokenization (both with trained Transformers). Sample-level is equivalent to Trilobyte at 8-bit and intractable at 24-bit (16.7M vocabulary). Transfer denotes a single 24-bit Trilobyte model trained on all datasets with lower bit masking for lower bit depths. Bold indicates best performance.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: FLAC compression performance across diverse audio domains at 8-bit and 16-bit quantization levels. Birdvox achieves exceptional compression (

\sim

6x at 8-bit), perhaps reflecting the sparse and structurally constrained nature of bird vocalizations, which are highly predictable under linear predictive coding. Meanwhile, speech and music datasets show more modest gains. 16-bit audio generally achieves 1.5–2.5x compression, with diminishing returns beyond FLAC level 3. Note that we disable FLAC’s verbatim, constant, and fixed subframe types, and that we do not evaluate Beethoven, YouTube Mix, or SC09 beyond 8-bit because they are 8-bit datasets.

Figure 3: Compression rate comparison across FLAC, DAC, EnCodec, and Custom DAC compressors on MusDB18 mixes. FLAC achieves the best compression, at approximately 1.8x, while the NAC-based approaches underperform, with EnCodec actually increasing file size.

Figure 4: Residual distribution comparison showing residual magnitudes (note the log scale) for FLAC, DAC, EnCodec, and Custom DAC compressors. FLAC residuals follow a geometric distribution with a mean absolute residual of 156.34, while DAC, EnCodec, and Custom DAC residuals are more uniformly distributed regardless of codebook level, with mean absolute residuals of 1,603.54 (DAC), 18,376.66 (EnCodec), and 1,245.76 (Custom DAC) – an order of magnitude larger than FLAC.

Figure 5: In-context LM-based compression performance with the method defined in Delétang et al. [ 5 ] and Li et al. [ 8 ] using pre-trained language models (Llama-2-7B and Llama-2-13B [ 15 ] ) across diverse audio domains at 8-bit and 16-bit quantization. We also report FLAC compression results at compression level 8, the maximum. Model scaling (7B to 13B) shows minimal gains at 8-bit and some improvements at 16-bit, especially for complex datasets. This method underperforms FLAC on most signals, with the exception of SC09 and Epidemic Sound at 8-bit.

Lossless Data Compression

See all 26 papers

Audio Token Attention Is Predictable Before the Language Model Runs
Sep 30, 2026Kyoungjun Park, Yunzhe Li, Lili QiuAudio Token Compression Audio Understanding
The University of Texas at Austin
AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs
Oct 9, 2026Shreya Sajid, Anna Vettoruzzo, Joaquin VanschorenAudio Token Compression Audio-Language Models
Department of Mathematics & Computer Science Eindhoven University of Technology Eindhoven, Netherlands
Locality Matters for Training-Free Audio Token Compression in Audio-Language Models
May 24, 2026Jiale Luo, Xiaoyu Liang, Haoji HuToken Merging Audio Token Compression
1Zhejiang University

Sep 30, 2026J/K move · Enter open · S save

Audio Token Attention Is Predictable Before the Language Model Runs

Kyoungjun Park, Yunzhe Li, Lili Qiu

The University of Texas at Austin

Audio Token Compression Audio Understanding Audio-Language Models Token Pruning LLM Inference Acceleration Token Compression

Open paper AskPDF arXiv

A large audio language model (LALM) turns a minute of speech into 750-1,500 tokens and prefills every one. Image-token pruning often cuts after the language model's first layers, where image tokens draw little attention. Audio tokens draw much more attention there, and their ranking is still far from final, so audio needs a ranking before the language model runs. Surprisingly, the attention an audio token will receive across the language model is already linearly predictable from its encoder output, before the language model runs. A linear map, fitted in closed form without labels, predicts this all-layer attention ranking at

ρ\geq .69

on eleven of thirteen LALMs. Our method, Triage, cuts audio tokens by this prediction and, on multiple choice, cuts again at layer 2, correcting the prediction with the attention observed there. Triage sets its compression without labels, under two budgets that limit how far its output may differ from the model's own full-audio output. At the conservative budget, its word error rate and accuracy stay within .04 of full audio. At the aggressive budget, Triage beats every baseline in all twelve transcription cases. On multiple choice, at 2.2-5x compression, it outperforms DART, the strongest baseline on average, by .043 in mean accuracy. Because it cuts before the language model, it raises the audio that fits in Qwen2.5-Omni-3B's context window from 21.8 to about 62 minutes. At its most compressive point, Triage lets one GPU serve 4x as many concurrent 5-minute streams of that model. Project page: https://audio-triage.github.io

Abstract

Lossless Data Compression

Explore similar work

Audio Token Attention Is Predictable Before the Language Model Runs

AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

Locality Matters for Training-Free Audio Token Compression in Audio-Language Models