Encode Once, Decode Never: Reusing Audio LM Internals for Efficient Temporal Localization

Joseph An; Phillip Keung; Jiaqi Wang; Orevaoghene Ahia; Noah A. Smith

cs.LGFeb 10, 2026

Encode Once, Decode Never: Reusing Audio LM Internals for Efficient Temporal Localization

Authors: Joseph An, Phillip Keung, Jiaqi Wang, Orevaoghene Ahia, Noah A. Smith

Organizations: University of Washington · Allen Institute for Artificial Intelligence

Abstract

Audio language models process input audio into rich frame-level representations, but the standard approach to temporal localization generates timestamps as sequences of text tokens, which discards the frame-level representations in favor of autoregressive decoding. However, generating timestamps as tokens is slow and not parallelizable, and tends to hallucinate when producing timestamps outside the training distribution. We propose internal frame-level reuse, a method that trains audio LMs to reuse their own internal audio representations for temporal localization directly, bypassing token generation altogether. We introduce a lightweight prediction head trained via different frame-level objectives: a binary frame classifier and a novel inhomogeneous Poisson process (IHP) loss that models temporal event intensity. Across word localization, speaker diarization, and event localization tasks, our approach can achieve a >50x inference speedup over token-based generation and demonstrates robust length generalization, maintaining high accuracy on out-of-distribution audio durations where token-based models collapse completely. We find that reusing audio frame-level representations yields localization accuracies comparable to (and often better than) finetuned token-based baselines.

Figures & tables

Panel of Figure 1 — Figure 1: Frame-level prediction versus standard autoregressive generation for predicting timestamps in audio. At inference time, we extract the predicted timestamp from the binary frame-level probabilities $p_{1},..,p_{T}$ via argmax.

Panel of Figure 2 — Figure 2: Inhomogeneous Poisson process frame-level training. In the simple case where only 1 timestamp is needed, the $\log\lambda_{1},...,\log\lambda_{T}$ are used to construct a non-parametric probability density estimator over the audio frames, normalized by the total intensity $\Lambda(T)=\sum_{k=1}^{T}\lambda_{k}$ .

Figure 3: Speed-up factor between token-based and Poisson-based timestamp generation across batch sizes. Efficiency gains are larger when the timestamp sequences are long. Poisson frame-level timestamps are as much

60\times

faster than token-based timestamps.

Train	Test	Tokens-only			Poisson			Binary
Train	Test	20ms $\uparrow$	40ms $\uparrow$	MAD $\downarrow$	20ms $\uparrow$	40ms $\uparrow$	MAD $\downarrow$	20ms $\uparrow$	40ms $\uparrow$	MAD $\downarrow$
Matched Distributions
0-4s	0-4s	93.5%	97.5%	0.01	94.1%	98.0%	0.01	85.4%	96.3%	0.03
0-8s	0-8s	93.7%	97.6%	0.01	93.9%	98.1%	0.01	88.2%	96.9%	0.02
0-12s	0-12s	93.5%	97.8%	0.01	94.1%	98.3%	0.01	86.6%	95.9%	0.03
0-16s	0-16s	93.7%	97.9%	0.01	93.9%	98.4%	0.01	85.3%	96.2%	0.04

Table 1: Librispeech word alignment accuracies and mean absolute deviation (MAD, in seconds) under matched (e.g., train on timestamps between 0–8 seconds, test on timestamps that are between 0–8 seconds) and mismatched (e.g., train on 0-8 seconds but test on 8–12 seconds) settings. The Poisson loss generalizes to out-of-distribution timestamps, unlike the token loss.

		Librispeech			Libricount			AudioSet
Model	Method	20ms $\uparrow$	40ms $\uparrow$	MAD $\downarrow$	40ms $\uparrow$	100ms $\uparrow$	MAD $\downarrow$	40ms $\uparrow$	100ms $\uparrow$	MAD $\downarrow$
Qwen2.5 3B	Tokens	91.4%	96.0%	0.08	60.8%	73.2%	0.13	40.9%	55.1%	1.15
	Poisson	93.8%	97.9%	0.01	64.8%	75.2%	0.13	41.0%	59.3%	1.02
	Binary	85.8%	96.1%	0.02	58.4%	72.0%	0.17	41.0%	57.7%	1.08
Qwen2.5 7B	Tokens	93.2%	97.1%	0.03	62.1%	75.3%	0.13	40.1%	56.9%	1.03
	Poisson	93.3%	98.1%	0.01	65.9%	76.5%	0.12	42.3%	58.5%	0.99

Table 2: Finetuned accuracies (Acc, %) and MAD (seconds) for Qwen audio LLMs with and without frame-level reuse. Highest accuracies and lowest MADs are bolded. Models tuned with Poisson loss outperform models tuned with either token or binary loss.

# of Words	Tokens-only			Poisson
# of Words	20ms $\uparrow$	40ms $\uparrow$	MAD $\downarrow$	20ms $\uparrow$	40ms $\uparrow$	MAD $\downarrow$
1-5	94.6%	98.1%	0.01	94.0%	97.4%	0.01
6-10	94.2%	97.3%	0.01	94.9%	97.8%	0.01
11-15	94.7%	97.8%	0.01	95.3%	98.1%	0.01
16-20	94.3%	97.3%	0.01	94.5%	97.1%	0.01
21-25	94.5%	97.6%	0.01	94.4%	97.3%	0.01

Table 3: Qwen2.5-Omni 7B performance on predicting start-of-word timestamps in Librispeech, stratified by number of words in the transcript. Training with either the token or Poisson loss result in very similar performance.

	Librispeech (Word Align.)			Libricount (Speaker Diar.)			Audioset (Human Vocal.)
Model	20ms $\uparrow$	40ms $\uparrow$	MAD $\downarrow$	40ms $\uparrow$	100ms $\uparrow$	MAD $\downarrow$	40ms $\uparrow$	100ms $\uparrow$	MAD $\downarrow$
Audio Flamingo 3	0.6%	1.4%	1.82	2.3%	6.3%	1.53	5.8%	10.2%	1.69
Voxtral 3B	0.8%	1.1%	2.52	0.3%	1.2%	6.02	1.2%	2.2%	3.74
Voxtral 24B	0.2%	0.5%	2.12	0.4%	1.4%	4.91	2.3%	4.2%	4.03
Qwen2.5 3B	0.1%	0.2%	2.59	0.6%	1.9%	3.71	3.6%	4.5%	4.46
Qwen2.5 7B	0.7%	1.1%	2.17	0.4%	1.5%	5.29	3.9%	5.7%	3.52

Table 4: Zero-shot performance of audio LMs on temporal tasks. We report accuracies (%) at different tolerances (20ms, 40ms, 100ms) and mean absolute deviation (MAD, seconds). Models are prompted to generate timestamps directly from audio without task-specific fine-tuning. Results show that all models struggle in this zero-shot setting, with low accuracy and high MAD across tasks.

	Librispeech			Libricount			AudioSet
	20ms $\uparrow$	40ms $\uparrow$	MAD $\downarrow$	40ms $\uparrow$	100ms $\uparrow$	MAD $\downarrow$	40ms $\uparrow$	100ms $\uparrow$	MAD $\downarrow$
Tokens-only	91.4%	96.0%	0.08	60.8%	73.2%	0.13	40.9%	55.1%	1.15
Poisson	93.8%	97.9%	0.01	64.8%	75.2%	0.13	41.0%	59.3%	1.02
Interpolated loss with Poisson frame-level loss and token loss
Token Output	92.2%	96.4%	0.06	62.8%	72.7%	0.15	41.2%	57.5%	1.01
Poisson Output	94.2%	98.1%	0.01	63.7%	74.0%	0.15	41.3%	58.5%	1.04

Table 5: Comparison of Qwen2.5-Omni 3B model performance between using token loss alone, Poisson loss alone, and an interpolation of the two losses. We ran all of our experiments with

L_{\text{interpolated}}=L_{\text{tokens}}+0.05L_{\text{Poisson}}

. Interpolating the two losses shows no consistent improvement on token outputs.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Learning Rate	$1\times 10^{-6}$
Effective Batch Size	8
Optimizer	AdamW
Adam $\beta_{1},\beta_{2}$	0.9, 0.999 (default)
Weight Decay	0.01 (default)
Training Epochs

Appendix

Table 6: Hyperparameter settings and training configurations.

Models	Sound	Music	Speech	Average
Qwen2-Audio-Instruct 8B	45.9	53.3	45.9	52.5
Qwen2.5-Omni 7B (Base Model)	64.7	61.2	64.1	63.3
Qwen2.5-Omni 7B (Poisson-Finetuned)	63.9	56.9	61.4	60.7

Appendix

Table 7: MMAU results on Sound, Music, Speech, and their average on the full test set.

	Librispeech			Libricount			AudioSet
Model	20ms $\uparrow$	40ms $\uparrow$	MAD $\downarrow$	40ms $\uparrow$	100ms $\uparrow$	MAD $\downarrow$	40ms $\uparrow$	100ms $\uparrow$	MAD $\downarrow$
3B	0.08	0.11	2.33	0.18	0.23	1.55	0.05	0.11	2.97
7B	0.04	0.06	2.21	0.14	0.19	1.79	0.10	0.19	2.74

Appendix

Table 8: Performance of the 3B and 7B models on Librispeech, Libricount, and AudioSet at different temporal resolutions when the audio LM is frozen and only the linear prediction head is trained.

Figure 4: Multi-timestamp decoding with our iterative mode finding and smoothing heuristic.

Temporal Audio Grounding

Temporal audio grounding locates when events, speech, or other audio evidence occur and links those intervals to language.

See all 36 papers

Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception
Sep 14, 2026Yanfeng Shi, Yan Song, Junhui Li +4Audio Understanding Audio-Language Models
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China · ICT Cluster, Singapore Institute of Technology, Singapore
TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models
Aug 30, 2026Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh +3Audio Understanding Audio-Language Models
University of Maryland, College Park, USA · CNH Industrial India
Locality Matters for Training-Free Audio Token Compression in Audio-Language Models
May 24, 2026Jiale Luo, Xiaoyu Liang, Haoji HuToken Merging Audio Token Compression
1Zhejiang University

Sep 14, 2026J/K move · Enter open · S save

Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception

Yanfeng Shi, Yan Song, Junhui Li, Tinggan Huang+3

National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China · ICT Cluster, Singapore Institute of Technology, Singapore

Audio Understanding Audio-Language Models Sound Event Detection Temporal Audio Grounding

Open paper AskPDF arXiv

Large Audio-Language Models (LALMs) have substantially advanced general audio understanding, yet they remain limited in fine-grained temporal perception, particularly in precise event localization. Existing approaches primarily post-train LALMs to predict event boundaries as timestamp tokens. However, this generative formulation lacks explicit correspondence between the timestamp predictions and fine-grained acoustic evidence, limiting the precision and reliability of temporal localization. To address this issue, we augment the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query. Specifically, the frozen LALM encodes the event query with audio as context, and the grounding model combines these query representations with fine-grained audio features to localize the target event at the frame level. Extensive experiments across diverse temporal grounding benchmarks demonstrate strong and consistent improvements over existing methods. Further evaluation shows that the grounding model can provide temporal evidence to support downstream reasoning.

Abstract

Temporal Audio Grounding

Explore similar work

Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

Locality Matters for Training-Free Audio Token Compression in Audio-Language Models