Organizations: Xi’an Jiaotong Liverpool University, China · MiLM Plus, Xiaomi Inc., China · Center for Machine Vision and Signal Analysis, University of Oulu, Finland · Institute of Acoustics, Chinese Academy of Sciences
Conventional audio language models (ALMs) have made significant progress in achieving alignment between auditory and textual representations, including recent explorations in spatial audio. However, in daily spatial scenarios, they still cannot effectively process multi-event audio sequences. Current approaches primarily rely on coarse-grained contrastive learning with global auditory and textual features, lacking the resolution to distinguish multiple sequential events. To overcome these limitations, we propose CoSTALA-a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning. By constructing a multi-granularity hierarchical loss function system, we achieve explicit modeling of temporal dependencies, and successfully anchors individual acoustic events to preserve their semantic purity. Extensive experiments demonstrate that CoSTALA significantly establish a powerful new framework for spatio-temporal audio understanding.
Figures & tables
Figure 1: Overview of the CoSTALA framework for multi-grain spatio-temporal audio-language contrastive learning. The architecture is designed to process long-duration temporal spatial audio and corresponding spatio-temporal captions derived from isolated spatial events. A text encoder, a semantic audio encoder and a spatial audio encoder process these inputs to extract multi-level representations, which are jointly aligned through a suite of hierarchical loss functions. Specifically, a contrastive learning loss Lcl and a local alignment loss Llocal establish multi-grain spatial audio-text mappings, while a spatio-temporal loss Lst explicitly models chronological dependencies, and a feature consistency loss Lconsist preserves local semantic purity.
Modality
Notation
Description
Text
Tsem
Semantic embeddings derived from concatenated captions.
Tspaci
Spatial embeddings of directional captions for chunk i .
Tst
Unified spatio-temporal embeddings of spatial multi-event captions.
Audio
Ehardci
Joint spatio-semantic representations from isolated event Ci .
Esoftci
Joint spatio-semantic representations from temporally sliced global maps.
Table 1: Summary of key embeddings used in CoSTALA.
Figure 2: Detailed audio pipeline. A shared encoder processes isolated and composed events into hard ( Ehard ) and soft ( Esoft ) representations. Ehard anchors consistency learning ( Lconsist ). A temporal encoder extracts sequential features ( Etemp ) from Esoft , which are fused with Eglobal to yield Est .
Model / Loss Config
Global Spatio-Temporal Retrieval
Semantic-Only Retrieval
Text-to-Audio
Audio-to-Text
Text-to-Audio
Audio-to-Text
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
SALM [ 11 ]
4.77
14.37
21.27
4.57
14.28
21.23
-
-
-
-
-
-
T-CLAP [ 5 ]
5.66
16.84
24.88
5.92
16.71
24.71
-
-
-
-
-
-
CoSTALA ( Lcl )
5.84
17.68
26.07
6.58
18.49
26.01
1.87
7.29
12.44
1.61
5.23
8.30
CoSTALA ( Lcl+Lst )
7.16
17.45
24.52
6.96
17.78
23.84
1.70
6.95
11.57
0.91
3.96
6.81
Table 2: Ablation study of the proposed hierarchical loss functions on spatial retrieval performance. We report Recall@ K ( K∈{1,5,10} ) in percentage (%) for both Text-to-Audio and Audio-to-Text directions. The left section evaluates the global spatio-temporal features, while the right section assesses the pure auditory semantic representations.
Large Audio-Language Models (LALMs) have substantially advanced general audio understanding, yet they remain limited in fine-grained temporal perception, particularly in precise event localization. Existing approaches primarily post-train LALMs to predict event boundaries as timestamp tokens. However, this generative formulation lacks explicit correspondence between the timestamp predictions and fine-grained acoustic evidence, limiting the precision and reliability of temporal localization. To address this issue, we augment the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query. Specifically, the frozen LALM encodes the event query with audio as context, and the grounding model combines these query representations with fine-grained audio features to localize the target event at the frame level. Extensive experiments across diverse temporal grounding benchmarks demonstrate strong and consistent improvements over existing methods. Further evaluation shows that the grounding model can provide temporal evidence to support downstream reasoning.
Yanfeng Shi, Yan Song, Junhui Li +4
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China · ICT Cluster, Singapore Institute of Technology, Singapore
Large Audio-Language Models (LALMs) enable general audio understanding and demonstrate remarkable performance across various audio tasks. However, these models still face challenges in temporal perception (e.g., inferring event onset and offset), leading to limited utility in fine-grained scenarios. To address this issue, we propose Audio-Side Time Prompt and leverage Reinforcement Learning (RL) to develop the TimePro-RL framework for fine-grained temporal perception. Specifically, we encode timestamps as embeddings and interleave them within the audio feature sequence as temporal coordinates to prompt the model. Furthermore, we introduce RL following Supervised Fine-Tuning (SFT) to directly optimize temporal alignment performance. Experiments demonstrate that TimePro-RL achieves significant performance gains across a range of audio temporal tasks, such as audio grounding, sound event detection, and dense audio captioning, validating its robust effectiveness.
Yanfeng Shi, Pengfei Cai, Jun Liu +5
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China · ICT Cluster, Singapore Institute of Technology, Singapore
Large Audio-Language Models (ALMs) have recently demonstrated remarkable capabilities in holistic audio understanding, yet they remain unreliable for temporal grounding, i.e., the task of pinpointing exactly when an event occurs within long-form audio. This limitation stems from two factors: training data dominated by clip-level supervision lacking precise timestamps, and benchmarks that fail to simulate real-world scenarios where short events are obscured by dense background sounds. In this paper, we introduce SpotSound, an audio language model designed for grounding audio events. SpotSound incorporates a novel training objective, specifically designed to suppress hallucinated timestamps for events absent from the input. Additionally, we present SpotSound-Bench, a challenging temporal grounding benchmark where target events occupy less than ~10% of each clip, creating a rigorous 'needle-in-a-haystack' evaluation. Experiments demonstrate that SpotSound achieves state-of-the-art results on temporal grounding benchmarks while maintaining robust performance across general downstream audio-language tasks. Code, models and benchmark are released on https://loiesun.github.io/spotsound/
Luoyi Sun, Xiao Zhou, Zeqian Li +3
Zhejiang University Hangzhou, Zhejiang, China · SAI, Shanghai Jiao Tong University Shanghai, China