CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning

Peiwei Ren; Jinbo Hu; Fang Kang; Shan Liang; Yin Cao

cs.SDAug 25, 2026

CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning

Authors: Peiwei Ren, Jinbo Hu, Fang Kang, Shan Liang, Yin Cao

Organizations: Xi’an Jiaotong Liverpool University, China · MiLM Plus, Xiaomi Inc., China · Center for Machine Vision and Signal Analysis, University of Oulu, Finland · Institute of Acoustics, Chinese Academy of Sciences

Abstract

Conventional audio language models (ALMs) have made significant progress in achieving alignment between auditory and textual representations, including recent explorations in spatial audio. However, in daily spatial scenarios, they still cannot effectively process multi-event audio sequences. Current approaches primarily rely on coarse-grained contrastive learning with global auditory and textual features, lacking the resolution to distinguish multiple sequential events. To overcome these limitations, we propose CoSTALA-a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning. By constructing a multi-granularity hierarchical loss function system, we achieve explicit modeling of temporal dependencies, and successfully anchors individual acoustic events to preserve their semantic purity. Extensive experiments demonstrate that CoSTALA significantly establish a powerful new framework for spatio-temporal audio understanding.

Figures & tables

Figure 1: Overview of the CoSTALA framework for multi-grain spatio-temporal audio-language contrastive learning. The architecture is designed to process long-duration temporal spatial audio and corresponding spatio-temporal captions derived from isolated spatial events. A text encoder, a semantic audio encoder and a spatial audio encoder process these inputs to extract multi-level representations, which are jointly aligned through a suite of hierarchical loss functions. Specifically, a contrastive learning loss

\mathcal{L}_{\mathrm{cl}}

and a local alignment loss

\mathcal{L}_{\mathrm{local}}

establish multi-grain spatial audio-text mappings, while a spatio-temporal loss

\mathcal{L}_{\mathrm{st}}

explicitly models chronological dependencies, and a feature consistency loss

\mathcal{L}_{\mathrm{consist}}

preserves local semantic purity.

Modality	Notation	Description
Text	$T_{\mathrm{sem}}$	Semantic embeddings derived from concatenated captions.
	$T_{\mathrm{spa}}^{\mathrm{c}i}$	Spatial embeddings of directional captions for chunk $i$ .
	$T_{\mathrm{st}}$	Unified spatio-temporal embeddings of spatial multi-event captions.
Audio	$E_{\mathrm{hard}}^{\mathrm{c}i}$	Joint spatio-semantic representations from isolated event $C_{i}$ .
	$E_{\mathrm{soft}}^{\mathrm{c}i}$	Joint spatio-semantic representations from temporally sliced global maps.
	$E_{\mathrm{temp}}$	Temporal representations capturing sequential dependencies.

Table 1: Summary of key embeddings used in CoSTALA.

Figure 2: Detailed audio pipeline. A shared encoder processes isolated and composed events into hard (

E_{\mathrm{hard}}

) and soft (

E_{\mathrm{soft}}

) representations.

E_{\mathrm{hard}}

anchors consistency learning (

\mathcal{L}_{\mathrm{consist}}

). A temporal encoder extracts sequential features (

E_{\mathrm{temp}}

) from

E_{\mathrm{soft}}

, which are fused with

E_{\mathrm{global}}

to yield

E_{\mathrm{st}}

Model / Loss Config	Global Spatio-Temporal Retrieval						Semantic-Only Retrieval
	Text-to-Audio			Audio-to-Text			Text-to-Audio			Audio-to-Text
	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
SALM [ 11 ]	4.77	14.37	21.27	4.57	14.28	21.23	-	-	-	-	-	-
T-CLAP [ 5 ]	5.66	16.84	24.88	5.92	16.71	24.71	-	-	-	-	-	-
CoSTALA ( $\mathcal{L}_{\mathrm{cl}}$ )	5.84	17.68	26.07	6.58	18.49	26.01	1.87	7.29	12.44	1.61	5.23	8.30
CoSTALA ( $\mathcal{L}_{\mathrm{cl}}+\mathcal{L}_{\mathrm{st}}$ )	7.16	17.45	24.52	6.96	17.78	23.84	1.70	6.95	11.57	0.91	3.96	6.81

Table 2: Ablation study of the proposed hierarchical loss functions on spatial retrieval performance. We report Recall@

K

(

K\in\{1,5,10\}

) in percentage (%) for both Text-to-Audio and Audio-to-Text directions. The left section evaluates the global spatio-temporal features, while the right section assesses the pure auditory semantic representations.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception
Sep 14, 2026Yanfeng Shi, Yan Song, Junhui Li +4Audio Understanding Audio-Language Models
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China · ICT Cluster, Singapore Institute of Technology, Singapore
Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt
Apr 15, 2026Yanfeng Shi, Pengfei Cai, Jun Liu +5Audio-Language Models Temporal Audio Grounding
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China · ICT Cluster, Singapore Institute of Technology, Singapore
SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding
Apr 14, 2026Luoyi Sun, Xiao Zhou, Zeqian Li +3Temporal Audio Grounding Audio-Language Models
Zhejiang University Hangzhou, Zhejiang, China · SAI, Shanghai Jiao Tong University Shanghai, China

Sep 14, 2026J/K move · Enter open · S save

Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception

Yanfeng Shi, Yan Song, Junhui Li, Tinggan Huang+3

National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China · ICT Cluster, Singapore Institute of Technology, Singapore

Audio Understanding Audio-Language Models Sound Event Detection Temporal Audio Grounding

Open paper AskPDF arXiv

Large Audio-Language Models (LALMs) have substantially advanced general audio understanding, yet they remain limited in fine-grained temporal perception, particularly in precise event localization. Existing approaches primarily post-train LALMs to predict event boundaries as timestamp tokens. However, this generative formulation lacks explicit correspondence between the timestamp predictions and fine-grained acoustic evidence, limiting the precision and reliability of temporal localization. To address this issue, we augment the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query. Specifically, the frozen LALM encodes the event query with audio as context, and the grounding model combines these query representations with fine-grained audio features to localize the target event at the frame level. Extensive experiments across diverse temporal grounding benchmarks demonstrate strong and consistent improvements over existing methods. Further evaluation shows that the grounding model can provide temporal evidence to support downstream reasoning.

Abstract

Audio-Language Models

Explore similar work

Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception

Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt

SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding