Organizations: X-LANCE Lab, Shanghai Jiao Tong University, China · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China · The Chinese University of Hong Kong, China
Contrastively pretrained audio-language models (e.g., CLAP) excel at clip-level understanding but struggle with frame-level tasks. Existing extensions fail to exploit the varying granularity of real-world audio-text data, where massive clip-level textual descriptions coexist with limited frame-level annotations. This paper proposes Fine-grained Language-Audio Pretraining (FineLAP), a novel training paradigm that advances both clip- and frame-level alignment in CLAP with heterogeneous data. FineLAP introduces a dual-stream sigmoid loss with a cluster-based sampling strategy to jointly learn from clip- and frame-level supervision. To capture both global semantics and local details, FineLAP uses a decoupled audio projector on top of a self-supervised encoder. To alleviate the scarcity of temporally annotated data, we present FineLAP-100k, a large-scale synthetic SED dataset constructed through a scalable curation pipeline. Extensive experiments demonstrate that FineLAP achieves SOTA performance across multiple audio understanding tasks, including retrieval, classification, sound event detection, and text-to-audio grounding. Ablation studies further show that coarse- and fine-grained alignment are mutually beneficial, providing insights for building better audio-language models (ALMs).
Figures & tables
Figure 1: FineLAP’s performance across different benchmark datasets: Our model achieves state-of-the-art (SOTA) results on both clip- and frame-level tasks.
Figure 2: FineLAP effectively leverages the fine-grained alignment between audio frames and textual phrases to detect temporal boundaries of overlapping events.
Figure 3: Overview of FineLAP. Left: The decoupled adapter enables simultaneous clip- and frame-level understanding. Right: The framework synergizes global and dense alignment by leveraging heterogeneous supervision via a dual-stream sigmoid loss with cluster-based sampling.
Methods
Type
Data †
Supervision ‡
LAION-CLAP Wu et al. (2023)
CLAP
2.7M
Global
HTSAT-BERT Mei et al. (2024)
CLAP
0.5M
Global
Cacophony Zhu et al. (2024)
CLAP
4.1M
Global
M2D-CLAP Niizumi et al. (2025)
CLAP
2.6M
Global
MGA-CLAP Li et al. (2024b)
CLAP
0.5M
Global
FLAM Wu et al. (2025)
CLAP
2.2M
Global + Frame
Table 1: Overview of baseline models. † : Data denotes number of audio clips used for training. ‡ : Supervision indicates whether training relies on clip-level annotations (Global) or frame-level annotations (Frame).
Methods
AudioCaps Eval (%)
Clotho Eval (%)
T–A Retrieval
A–T Retrieval
T–A Retrieval
A–T Retrieval
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
R@1
R@5
R@10
LAION-CLAP Wu et al. (2023)
35.1
71.9
83.7
44.2
80.8
90.3
16.9
41.6
54.4
24.4
49.3
65.7
HTSAT-BERT Mei et al. (2024)
39.7
74.5
86.1
51.7
82.3
90.6
19.5
45.2
58.2
23.4
50.9
63.4
Cacophony Zhu et al. (2024)
41.0
75.3
86.4
55.3
83.6
92.4
20.2
45.9
58.8
26.5
54.1
67.3
MGA-CLAP Li et al. (2024b)
42.2
74.9
-
53.7
84.3
-
20.8
45.0
-
26.5
54.1
-
Table 2: Audio-text retrieval performance on the evaluation split of AudioCaps and Clotho.
Methods
DESED
AS-SL
USED
TAG
HTSAT-BERT Mei et al. (2024)
0.131
0.284
0.016
0.344
MGA-CLAP Li et al. (2024b)
0.264
0.354
0.087
0.487
FLAM Wu et al. (2025)
0.094
0.351
0.295
-
FlexSED Hai et al. (2025)
0.161
0.448
0.052
-
PE A-Frame Vyas et al. (2025)
0.344
0.431
0.123
-
PT-SED Schmid et al. (2025)
-
0.465
-
-
Table 3: Sound event detection performance on DESED, AudioSet-Strong (AS-SL), UrbanSED (USED) and AudioGrounding (TAG).
Methods
ESC-50
US8K
VGGSound
LAION-CLAP Wu et al. (2023)
91.0
77.0
29.1
HTSAT-BERT Mei et al. (2024)
94.8
80.6
29.6
Cacophony Zhu et al. (2024)
93.4
77.1
27.0
MGA-CLAP Li et al. (2024b)
94.9
83.7
31.8
FLAM Wu et al. (2025)
86.9
75.6
39.3
M2D-CLAP Niizumi et al. (2025)
94.3
82.3
-
Table 4: Audio classification performance on ESC-50, UrbanSound8K, and VGGSound.
Methods
Audio-Text Retrieval
Sound Event Detection
Audio Classification
AC-T2A ∗
AC-A2T ∗
DESED
AS-SL
UrbanSED
TAG
VGGSound
US8K
FineLAP (Ours)
45.7
62.5
0.344
0.474
0.446
0.649
31.5
84.9
w/o Frame-level Loss
44.4
58.3
0.021
0.191
0.000
0.303
31.4
82.7
w/o Clip-level Loss
4.9
6.2
0.322
0.451
0.462
0.480
11.3
58.6
w/o Decoupled Projector
44.8
55.4
0.317
0.467
0.442
0.448
31.8
85.4
w/o Synthetic Data
45.4
61.6
0.324
0.468
0.154
0.589
31.3
78.0
Table 5: Ablation studies on loss design, model architecture and synthetic data. ∗ : AC-T2A and AC-A2T denote the R@1 retrieval performance on the evaluation split of AudioCaps.
Methods
Fine-grained Projector
Audio-Text Retrieval
Sound Event Detection
Audio Classification
AC-T2A
AC-A2T
DESED
AS-SL
UrbanSED
TAG
VGGSound
US8K
FineLAP (Ours)
Transformer
45.7
62.5
0.344
0.474
0.446
0.649
31.5
84.9
BiGRU
45.8
61.7
0.329
0.471
0.432
0.589
31.1
84.5
Linear
46.3
61.1
0.329
0.463
0.435
0.570
31.5
83.0
Table 6: Ablation studies on the architecture of fine-grained audio adapter fa .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Nphrases
DESED-PSDS ↑
10
0.163
15
0.175
20
0.182
25
0.184
30
0.181
Appendix
Table 7: Sensitivity analysis on the number of phrases N in the clustering-based negative sampling design.
Figure 4: Visualization of FineLAP’s performance on DESED-Eval
Figure 5: Visualization of FineLAP’s performance on AudioSet-Strong
Figure 6: Visualization of the proposed window-based audio clipping strategy
Xi’an Jiaotong Liverpool University, China · MiLM Plus, Xiaomi Inc., China · Center for Machine Vision and Signal Analysis, University of Oulu, Finland +1