Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt

Yanfeng Shi; Pengfei Cai; Jun Liu; Qing Gu; Nan Jiang; Lirong Dai; Ian McLoughlin; Yan Song

cs.SDApr 15, 2026

Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt

Authors: Yanfeng Shi, Pengfei Cai, Jun Liu, Qing Gu, Nan Jiang, Lirong Dai, Ian McLoughlin, Yan Song

Organizations: National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China · ICT Cluster, Singapore Institute of Technology, Singapore

Abstract

Large Audio-Language Models (LALMs) enable general audio understanding and demonstrate remarkable performance across various audio tasks. However, these models still face challenges in temporal perception (e.g., inferring event onset and offset), leading to limited utility in fine-grained scenarios. To address this issue, we propose Audio-Side Time Prompt and leverage Reinforcement Learning (RL) to develop the TimePro-RL framework for fine-grained temporal perception. Specifically, we encode timestamps as embeddings and interleave them within the audio feature sequence as temporal coordinates to prompt the model. Furthermore, we introduce RL following Supervised Fine-Tuning (SFT) to directly optimize temporal alignment performance. Experiments demonstrate that TimePro-RL achieves significant performance gains across a range of audio temporal tasks, such as audio grounding, sound event detection, and dense audio captioning, validating its robust effectiveness.

Figures & tables

Figure 1: Overview of our TimePro-RL framework. Timestamp Embeddings are interleaved within audio features as time prompt, followed by SFT and RL post-training.

Zero-shot
Model	Scale	Audio Grounding				Sound Event Detection	Dense Audio Captioning
Model	Scale	[email protected]	[email protected]	[email protected]	mIoU	Eb-F1	METEOR	Eb-F1
Qwen2-Audio	7B	9.2	5.1	3.3	11.9	3.4	11.2	3.0
Qwen2.5-Omni	7B	25.4	17.4	10.6	27.7	13.7	10.5	10.4
Finetuned
Audio-Flaming2	3B	37.0	27.6	19.0	43.3	8.9	25.7	12.7

Table 1: Performance comparison of TimePro-RL and zero-shot / finetuned baseline models on audio temporal tasks. Bold indicates the best results in each category.

Task	Training Size	Test Size
Audio Grounding	61,862	483
Sound Event Detection	15,041	1,153
Dense Audio Captioning	92,443	741

Table 2: Summary of data statistics across the three tasks.

Method	Audio Grounding				Sound Event Detection	Dense Audio Captioning
(Qwen2.5-Omni)	[email protected]	[email protected]	[email protected]	mIoU	Eb-F1	METEOR	Eb-F1
SFT Baseline	74.0	59.8	34.1	69.9	48.9	31.3	35.2
w/ ASTP (random init)	73.2	57.2	32.8	68.8	46.0	31.4	33.3
w/ ASTP	77.6	61.7	35.8	71.7	50.1	32.6	37.0
w/ ASTP + RL (Eb-F1)	77.8	63.1	38.9	72.7	56.9	31.6	38.1
w/ ASTP + RL	80.1	66.3	39.8	74.4	57.6	33.9	40.7

Table 3: Ablation study of different components. ASTP denotes Audio-Side Time Prompt; “random init” refers to random initialization of Timestamp Embeddings; RL(Eb-F1) indicates using only Eb-F1 as the reward.

Figure 2: Visualization of attention weights on Timestamp Embeddings. For the audio grounding query “a train horn honking”, the mel-spectrogram (top) is aligned with the chronologically arranged attention weights assigned to each Timestamp Embedding (bottom).

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception
Sep 14, 2026Yanfeng Shi, Yan Song, Junhui Li +4Audio Understanding Audio-Language Models
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China · ICT Cluster, Singapore Institute of Technology, Singapore
TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models
Aug 30, 2026Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh +3Audio Understanding Audio-Language Models
University of Maryland, College Park, USA · CNH Industrial India
SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding
Apr 14, 2026Luoyi Sun, Xiao Zhou, Zeqian Li +3Temporal Audio Grounding Audio-Language Models
Zhejiang University Hangzhou, Zhejiang, China · SAI, Shanghai Jiao Tong University Shanghai, China

Sep 14, 2026J/K move · Enter open · S save

Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception

Yanfeng Shi, Yan Song, Junhui Li, Tinggan Huang+3

National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China · ICT Cluster, Singapore Institute of Technology, Singapore

Audio Understanding Audio-Language Models Sound Event Detection Temporal Audio Grounding

Open paper AskPDF arXiv

Large Audio-Language Models (LALMs) have substantially advanced general audio understanding, yet they remain limited in fine-grained temporal perception, particularly in precise event localization. Existing approaches primarily post-train LALMs to predict event boundaries as timestamp tokens. However, this generative formulation lacks explicit correspondence between the timestamp predictions and fine-grained acoustic evidence, limiting the precision and reliability of temporal localization. To address this issue, we augment the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query. Specifically, the frozen LALM encodes the event query with audio as context, and the grounding model combines these query representations with fine-grained audio features to localize the target event at the frame level. Extensive experiments across diverse temporal grounding benchmarks demonstrate strong and consistent improvements over existing methods. Further evaluation shows that the grounding model can provide temporal evidence to support downstream reasoning.

Abstract

Audio-Language Models

Explore similar work

Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding