LAION-Audio-XL: A Web-Scale Corpus and Scaling Studies for Contrastive Language-Audio Pretraining
Authors: Gijs Wijngaard, Felix Förster, Ross Wightman, Marianna Nezhurina, Christoph Schuhmann, Jenia Jitsev, A. Sophia Koepke
Organizations: Jülich Supercomputing Center (JSC), Forschungszentrum Jülich (FZJ) · LAION · Technical University of Munich, MCML · University of Tübingen, Tübingen AI Center
Recent progress on contrastive audio-language models can be attributed mainly to the scale and quality of their training data. Larger training datasets expose models to a broader range of sounds and corresponding audio captions. To exploit this, we collect 1.48 billion audio samples from academic datasets, internet datasets, web collections, Common Crawl, and YouTube, and process them with a multi-stage curation pipeline. Furthermore, we generate short and long captions for samples in our corpus that lack them. We use our dataset to study the scaling behavior of contrastive audio-language models based on the NaFlex architecture for the audio encoder. Our experiments identify trade-offs between model size and training data for a given compute budget. Moreover, our resulting models, CLAP2, outperform prior audio-text models of comparable size on the MAEB and HEAR benchmarks. We release our dataset, captions, model checkpoints, and code.
Figures & tables
Figure 1: Our training dataset consists of many sources, spanning diverse sound categories, and our models achieve strong performance that improves with training scale. Left : the 365M training audio samples from source group to predicted sound category using our trained taxonomy classifier (see Section 3.3 ). Ribbon width is proportional to audio samples. Middle : MAEB [ Assadi et al., 2026 ] of CLAP2-S, LAION Larger CLAP [ Wu et al., 2023 ] and MS-CLAP 2023 [ Elizalde et al., 2024 ] . Dotted: previous best below 500M parameters. Right : Scaling MAEB-30 against samples seen.
Dataset
Sources
Audio
Captions
Duration (h)
Clotho [ Drossos et al., 2020 ]
1
6K
30K
37 ∗
AudioCaps [ Kim et al., 2019 ]
1
51K
57K
143 ∗
WavCaps [ Mei et al., 2024 ]
4
403K
403K
7.6K
LAION-Audio-630K [ Wu et al., 2023 ]
8
634K
634K
4.3K
Auto-ACD [ Sun et al., 2024 ]
2
1.9M
1.9M
5.3K ∗
SALMONN-2 [ Yang et al., 2026b ]
21
–
8.9M
18.2K
Table 1: LAION-Audio-XL compared to existing audio-text datasets , including standalone dataset releases and the training data of audio encoders and audio-language models. Our LAION-Audio-XL dataset is sourced from a range of online platforms and websites, and from existing data collections, and datasets (details are provided in Appendix H ). † Dasheng trains on audio only. ‡ Not publicly released. Captions counts the text entries each source reports (captions, ASR transcripts, QA pairs); – under Audio means not reported; ∗ marks values derived from clip counts and clip lengths.
Figure 2: Overview of the LAION-Audio-XL dataset curation pipeline. We collected 1.48B audio samples from online platforms, of which 1.47B entered deduplication. Removing silent audio, duplicates (including SemDeDup [ Abbas et al., 2023 ] ), and evaluation data leaves 1.28B audio samples. We then selected 500M audio samples based on categorical taxonomy annotations. The same annotations were used to prompt Qwen3-Omni [ Xu et al., 2025b ] and MOSS-Audio [ Yang et al., 2026a ] to generate short and long captions for 471M audio samples without textual descriptions, resulting in 500M captioned audio samples in total (the 500M selection). The 365M audio samples that are up to 30 seconds long form LAION-Audio-XL.
MAEB
Domain
Task type
Model
Params
Mean
Speech
Music
Sound
Clf.
ZS
Pair
Clust.
Retr.
Large models ( ≥ 1B parameters)
LCO-Embedding-Omni-7B [ Xiao et al., 2025 ]
7.7B
53.5
54.8
73.7
29.7
57.9
64.5
67.3
1.7
57.6
e5-omni-7B [ Chen et al., 2026 ]
7.7B
52.5
54.1
71.8
28.1
53.6
68.8
65.5
1.8
59.1
BidirLM-Omni-2.5B [ Boizard et al., 2026 ]
2.0B
52.4
58.6
66.0
20.0
59.9
65.8
66.7
6.2
50.2
Table 2: MAEB benchmark results. Parameter counts (Params) include only the audio and text parts of each model. Results for our models use the last checkpoint of each size’s WSD run: 1024M samples for XXS, XS, S, M, L, evaluated with un-normalised audio embeddings and with all-caps text inputs lowercased. We report the mean performance over 30 tasks, domain-specific (18 speech, 6 music, 6 sound tasks), and task-specific results: classification (12, including one multilabel), zero-shot classification (2), pair classification (3), clustering (3), and retrieval (10, including one reranking task). All scores are in %. Models are grouped by size, the best value in each column within a group is bolded.
Parameters (M)
GFLOPs / clip
Tier
d
Audio
Text
Total
Audio
Text
Total
XXS
224
8.3
7.8
27.3
21.6
4.6
26.1
XS
320
16.6
15.7
48.3
38.0
8.8
46.8
S
512
41.8
36.4
103.9
83.8
19.6
103.4
M
640
64.4
56.2
152.8
123.1
29.8
152.9
L
960
166.5
125.4
340.1
292.8
65.1
357.9
Table 3: Model tiers for scaling study: Hidden dimension d , parameter counts, and forward-pass compute per encoder. Audio parameters include the MAP pool and projection layers. Text parameters include the pool and output projection but exclude the 50,260×dt token-embedding table. Total parameters refer to the full model size, including embeddings. GFLOPs are forward-pass FLOPs per clip at the training length distribution.
Figure 3: Scaling behavior on our LAION-Audio-XL dataset. MAEB-30 error rate ( 1−mean score ) per compute C (GFLOPs). Left : the compute-Pareto-optimal runs (solid points) of all warmup-stable-decay and cosine learning-rate-scheduled runs, with our fitted scaling function and its 95% bootstrap CI (1000 resamples). With α=0.0574 , the error reduces by 3.9% for each doubling of compute ( ≈+1.8 mean MAEB-30 points at the largest budget). Right : each model size, from two WSD runs per size with individual cooldown phases, and the compute-Pareto envelope.
Figure 7
Figure 6: Embedding model performance by size. Average score across the individual tasks of MAEB (all 30 tasks) and HEAR (17 tasks). Blue squares are our own runs. Filled circles are audio-text models, and unfilled circles are audio-only models with no text component. Model sizes count only the audio and text parts of each model.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Segmentation comparison after 8M samples per run. Diamonds and whiskers show the MAEB-30 mean and standard deviation over three seeds. Open circles show individual runs. Time: seconds from reading the source audio to written FLAC clips, per hour of written clips, on the same 227 hours of YouTube audio using 72 CPU cores.
Figure 8: Visualization of the sound taxonomy. The five levels contain 11, 837, 3,150, 393, and 25 nodes, respectively, for a total of 4,416 nodes, including 4,085 leaves. Coloured inner arcs denote the 11 top-level groups, ordered clockwise from speech at the top. Branches show 77 of the 837 second-level categories, selected by descendant leaf count and audio sample count in the deduplicated pool of 1.28B audio samples (grey numbers, respectively). Italic rows summarize omitted categories and their leaves.
speed ↑
diversity ↑
quality
Captioner
samples/s per node
vocab @80k
vocab @400k
human rank ↓
LLM rank ↓
LLM BT ↑
MOSS-Audio-8B-Thinking (think-ON)
22.9
5,488
16,613
2.91 ± 0.39
2.76 ± 0.50
1.694
MOSS-Audio-8B-Thinking (think-OFF)
19.2
5,054
15,768
3.24 ± 0.36
2.61 ± 0.40
1.755
Qwen2.5-Omni-7B
23.0
4,652
8,660
4.10 ± 0.45
4.43 ± 0.50
0.873
AudioFlamingoNext
55.5
4,743
9,858
4.26 ± 0.41
4.51 ± 0.53
0.714
Qwen3-Omni-30B-A3B
75.9
2,693
6,619
4.32 ± 0.45
3.43 ± 0.51
1.093
Appendix
Table 4: Caption Model Comparison . Arrows indicate whether higher ( ↑ ) or lower ( ↓ ) is better. Speed: Throughput is given in samples/s per node (4x GH200 GPUs). Diversity: Vocab@80k and vocab@400k are the number of unique words across 80k short and 400k long caption tokens, respectively. Quality: Human rank and LLM rank (Gemini 3.6 Flash) report the mean rank among the 7 captioners for the 50 shared audio samples (1 = best, 7 = worst) ± 95% CI. LLM BT is the Bradley–Terry strength for the Gemini 3.6 Flash judge on the full 8997 test audio samples.
#
Model
Caption type
Mean rank ↓
Win rate (%) ↑
1
Qwen3-Omni
Raw JSON
3.20±0.10
19.9±2.5
2
MOSS Audio
Raw JSON
3.35±0.10
16.4±2.4
3
Qwen3-Omni
Natural sounding
3.46±0.10
14.7±2.2
4
MOSS Audio
Natural sounding
3.56±0.10
12.9±2.1
5
MOSS Audio
Audio only
3.58±0.12
19.6±2.5
6
Qwen3-Omni
Audio only
3.84±0.12
16.6±2.4
Appendix
Table 5: Attaching raw metadata strings leads to the highest accuracy. Overall ranking of two captioning models with three prompt types across the same 976 audio samples, evaluated with Gemini 3.6 Flash (thinking: low). Mean rank ranges from 1 (best) to 6 (worst). Win rate refers to being the best caption for a clip (higher-is-better). The ± values give 95% percentile confidence intervals over B=20,000 resamples of the 976 audio samples.
Tier
Audio
Text
d
A-Size
T-Size
A:T
N
Total
GFLOPs
da×La , h , dff
dt×Lt , h , dff
(M)
(M)
(M)
(M)
/clip
XXS
224×12 , 4, 640
224×12 , 4, 640
224
8.3
7.8
1.06
16.0
27.3
26.1
XS
320×12 , 5, 896
320×12 , 5, 896
320
16.6
15.7
1.06
32.2
48.3
46.8
S
512×12 , 8, 1408
512×11 , 8, 1408
512
41.8
36.4
1.15
78.2
103.9
103.4
M
640×12 , 10, 1728
640×11 , 10, 1728
640
64.4
56.2
1.15
120.6
152.8
152.9
L
1024×12 , 16, 2816
960×11 , 15, 2560
960
166.5
125.4
1.33
291.8
340.1
357.9
Appendix
Table 6: Model tiers for WSD scaling runs: Audio and text columns give width × depth, number of attention heads, and SwiGLU hidden size per encoder. A-Size and T-Size are the respective encoder parameter counts in millions. Audio counts include the MAP pool and projection layers. Text counts include the pool and output projection but exclude the 50,260×dt token-embedding table. N (non-embedding) is the sum of A-Size and T-Size. GFLOPs are total (audio + text tower) forward-pass FLOPs per clip at the training length distribution.
Audio front-end
Sample rate / mel bins
32 kHz / 128 (50 Hz–16 kHz)
Window / hop
1024 / 320 samples (100 frames/s)
Patch (freq × time)
16×16 (160 ms), 50 tokens/s
Max. clip length
30 s (1504 tokens), train and eval
Length buckets (tokens)
344 / 448 / 648 / 1128 / 1504
Audio tokens per GPU step
49,152
Appendix
Table 7: Architecture settings shared by all tiers in the scaling study. For details on architecture shape refer to Table 6 & 8 .
Figure 9: Audio:text capacity choice does not significantly affect performance up to 64M samples. Left: parameter composition of the four capacity splits for model sizes XS and M. Within a tier, the non-embedding total is held constant. The 50k-vocab token embedding is adapted to the individual layer dimensions and is a fixed lookup cost, not encoder capacity. Right: Mean MAEB percentage scores over 30 tasks per training sample seen D show that the audio-to-text tower ratio does not have a significant effect up to 64M samples.
Tier
Audio
Text
d
A-Size
T-Size
A:T
N
Total
GFLOPs
da×La , h , dff
dt×Lt , h , dff
(M)
(M)
(M)
(M)
/clip
XXS-C
256 × 12, 4, 704
192 × 12, 3, 512
192
10.5
5.5
1.92
15.9
25.6
29.2
XS-C
384 × 12, 6, 1024
256 × 12, 4, 682
256
22.9
9.7
2.36
32.6
45.5
58.7
S-C
576 × 12, 9, 1536
384 × 12, 6, 1024
384
51.5
21.8
2.36
73.3
92.6
117.1
M-C
768 × 12, 12, 2048
512 × 12, 8, 1365
512
91.4
38.8
2.36
130.2
156.0
194.9
L-C
896 × 21, 14, 2389
640 × 12, 10, 1706
640
211.3
60.6
3.49
271.9
304.1
431.1
Appendix
Table 8: Model tiers for cosine scheduled scaling runs. Audio and text columns give width × depth, number of attention heads, and SwiGLU hidden size per encoder. A-Size and T-Size are the respective encoder parameter counts in millions. Audio counts include the MAP pool and projection layers. Text counts include the pool and output projection but exclude the 50,260×dt token-embedding table. N (non-embedding) is the model size used in all scaling fits while total refers to the actual model size. GFLOPs are total (audio + text tower) forward-pass FLOPs per clip at the training length distribution.
Global Batch
Models
Samples
Warmup
Cooldown
Min
Max
Avg
LR
XXS–L
1M
350
224
512
2,176
916
4.0×10−4
XXS–L
2M
350
448
512
2,176
916
4.0×10−4
XXS–L
4M
350
896
512
2,176
916
4.0×10−4
XXS–L
8M
350
1,792
512
2,176
916
4.0×10−4
XXS–L
16M
350
3,584
512
2,176
916
4.0×10−4
Appendix
Table 9: Training configuration of the cooled-down WSD endpoints. “XXS–L” denotes all five of our model sizes (XXS, XS, S, M, L); Samples is the number of training samples seen. Every model size shares the schedule at a given budget. Warmup and cooldown are in optimizer steps, and the cooldown decays the learning rate linearly to zero over the final 20 % of the budget. Batch size varies based on the length buckets. L at 1,024M ran on twice the GPUs with half the per-GPU token budget (24,576), which gives the same global batches.
Global Batch
Models
Samples
Warmup
Nodes
Min
Max
Avg
LR
(XXS-L)-C
2M
472
2
256
1,088
461
2.5×10−4
(XXS-L)-C
4M
472
4
512
2,176
922
2.5×10−4
(XXS-L)-C
8M
943
4
512
2,176
922
5.0×10−4
(XXS-L)-C
16M
943
8
1,024
4,352
1,843
5.0×10−4
(XXS-L)-C
32M
1,183
12
1,536
6,528
2,765
1.0×10−3
Appendix
Table 10: Training hyperparameters for the cosine learning rate scheduled scaling runs. “(XXS-L)-C” denotes all five of our model sizes (XXS-C, XS-C, S-C, M-C, L-C); Samples is the number of training samples seen. Batch size varies based on the length buckets.
Figure 10: Scaling behavior on the full MAEB benchmark and per task type. Error rate ( 1−mean score ) of the full MAEB-30 and of each task-type subset per compute C (GFLOPs). The band marks the 95% bootstrap CI with 1000 resamples. It shows clear trends in compute scaling for zero-shot (ZS) classification, pair classification, and retrieval. In contrast, it isolates the saturating classification task.
Task
Type
XXS (27M)
XS (48M)
S (104M)
M (153M)
L (340M)
Samples seen
1024M
1024M
1024M
1024M
1024M
Speech (18 tasks)
SpokenSQuAD T2A
Retr.
34.3
39.0
38.0
41.7
45.0
CommonVoiceMini21 T2A
Retr.
60.8
65.6
68.0
71.9
70.7
FLEURS T2A
Retr.
47.3
50.7
55.2
61.8
59.4
GigaSpeech T2A
Retr.
59.1
64.3
71.6
70.4
72.3
Appendix
Table 11: MAEB-30 task scores (%) of the final scaling-grid models, each the cooled-down checkpoint of its tier at 1024M samples seen. Parameter counts are totals over both towers, including embeddings. All models are evaluated with native MTEB 2.21.0 [ Muennighoff et al., 2023b ] on the same embedding wrapper, with audio capped at 30 s (1504 tokens), un-normalised audio embeddings, and all-caps text inputs lowercased. Multi-subset tasks average all subsets. Bold marks the best value per row.
Figure 21
Samples
Variant
FLOPs ( 1020 )
MAEB-30
Speech
Music
Sound
Baseline
256M
up to 120 s
4.97
43.0
40.2
68.4
26.2
Audio length
256M
drop audio above 30 s
1.44
47.1
45.8
72.3
26.1
256M
crop to 30 s
1.76
46.5
45.1
70.6
26.4
256M
short-clip oversampling
3.18
41.9
39.0
67.3
24.8
Captions
128M
50% short, 50% long captions
2.48
39.1
34.9
66.3
24.6
128M
100% short captions
2.48
38.6
34.1
66.7
23.9
Appendix
Table 12: Impact of training on different audio lengths, and caption types on mean MAEB-30 performance for our model of size M-C. Domain columns average the 18 speech, 6 music, and 6 sound tasks. All models were trained on the 500M selection with a 120 second audio length limit unless stated otherwise.
Domain
Task type
Model
Params
Mean
Sound
Speech
Music
Clf.
Tagging
Events
Audio-only models
USAD 2.0-XXLarge+ [ Chang et al., 2026 ]
1.0B
86.8
74.7
90.8
92.6
87.9
62.5
95.0
USAD 2.0-XLarge+ [ Chang et al., 2026 ]
695M
86.8
74.7
90.6
92.7
87.8
62.8
95.3
AF3 Whisper encoder [ Goel et al., 2025 ]
637M
85.6
74.6
89.2
90.9
86.6
61.4
93.8
Dasheng-1.2B [ Dinkel et al., 2024b ]
1.2B
83.7
70.4
87.0
90.8
85.0
54.2
94.2
Appendix
Table 13: HEAR benchmark results for the models of Table 2 and the audio-only encoders of Figure 6 , with the same embeddings as there. We report the mean over 17 tasks (all HEAR tasks except Beehive and Maestro 5h), domain-specific means and the three HEAR task types: clip classification, multi-label tagging (FSD50K) and timestamp event detection (DCASE 2016). Audio-only models are evaluated by the models’ authors or the official HEAR 2021 leaderboard. Audio-text models are evaluated by us. Within each group models are sorted by the mean, and only models with every value available are listed. For audio-only models, we retain the best one or two variants of each method by mean score All scores are in %. The best listed value in each column is bolded, separately for the audio-only models and for the audio-text models together with ours.
Domain
Task type
Model
Params
Mean
Speech
Sound
Bird
Clf.
Clust.
Rerank.
Reason.
Retr.
Stab. ↓
M2D-CLAP [ Niizumi et al., 2024 ]
199M
30.7
36.4
52.5
17.1
34.6
50.6
1.4
5.5
0.2
9.4
LAION-CLAP HTSAT unfused [ Wu et al., 2023 ]
153M
30.0
36.0
58.6
15.3
33.9
49.3
1.6
6.1
0.0
11.8
MS-CLAP 2023 [ Elizalde et al., 2024 ]
160M
28.5
33.6
55.1
15.7
25.5
50.9
1.6
5.4
0.0
10.2
LAION Larger CLAP [ Wu et al., 2023 ]
194M
28.2
32.9
58.8
16.0
33.7
45.3
1.4
5.7
0.0
21.5
LCO-Embedding-Omni-7B [ Xiao et al., 2025 ]
7.7B
27.9
33.4
55.7
14.2
23.4
12.7
41.7
15.1
74.3
13.5
Appendix
Table 14: MSEB benchmark results on the English tasks, for most models of Table 2 (same parameter counts and checkpoints). We report the mean over 62 higher-is-better scores from 52 tasks (speaker clustering scores age, gender and speaker identity separately), domain-specific means (40 speech, 2 environmental sound, 20 bird sound scores) and task-specific means: classification (16), clustering (26), reranking (5), reasoning (5), and retrieval (10). Stability (Stab., 5 tasks) is the continuous edit distance between the embeddings of clean and noise-augmented speech; it is lower-is-better and not part of the mean. Speech transcription and salient-term segmentation are not reported, since they score text output that an embedding model does not produce. Multilingual tasks, the image–speech retrieval tasks, and the full-index document retrieval (we use the small index) are not run. All scores are in %. The best value in each column is bolded.
Archive
500M selection
LAION-Audio-XL
Source group
Sources
Audio
Hours
Audio
Hours
Audio
Hours
Academic datasets
316
83,689,086
231,011
32,815,997
82,820
31,283,585
57,789
Internet datasets
54
9,614,142
22,287
2,131,152
5,204
2,112,543
4,911
Web crawl and web collections
54
158,932,602
1,276,651
33,547,324
394,100
17,050,765
98,007
YouTube-derived data
1
1,179,581,779
9,420,695
430,719,261
2,865,365
315,559,153
1,228,570
Total
425
1,431,817,609
10,950,644
499,213,734
3,347,490
366,006,046
1,389,277
Appendix
Table 15: Sources, stored audio samples and indexed audio hours by group, at three stages: the archive as collected, the 500M selection, and LAION-Audio-XL (audio samples at most 30 seconds long); the last two count their training split. A clip re-cut from a recording above 120 seconds is charged once to its source in the last two stages. The archive counts the recording as collected. Rows count the sources listed in Table 16 . The deduplicated pool between the first two stages is given in Figure 2 .
Archive
500M selection
LAION-Audio-XL
Dataset
Audio
Hours
Audio
Hours
Audio
Hours
Academic datasets — 316 sources
4Q Audio Emotion [ Panda et al., 2020 ]
2,878
23.8
150
1.2
60
0.5
AAM [ Ostermann et al., 2023 ]
14,741
464.6
9,247
274.7
340
1.9
ACalt4 [ Niizumi et al., 2024 ]
120,464
330.0
1,433
3.9
1,433
3.9
ACM MIRUM [ Peeters and Flocon-Cholet, 2012 ]
1,312
13.5
1,261
12.9
728
6.1
Appendix
Table 16: Complete dataset catalogue, alphabetized inside each group. Archive counts every clip stored in the collection before deduplication. 500M selection counts the selected audio samples. LAION-Audio-XL counts the subset of audio samples of at most 30 seconds. Both count their training split. A source counts as academic if it is linked to a publication, in the broadest sense. Otherwise it counts as an internet source. Sources with fewer than 10 LAION-Audio-XL audio samples were removed from both corpora unless they are part of a listed source, whose row counts them, so every source of the two corpora is listed. Names link to source pages where the dataset is found, or in some cases the paper itself.
Audio-language pretraining (ALP) holds promise for learning general-purpose audio representation, yet remains underexplored. Crucially, there is no consensus on whether audio-language models can build effective general-purpose audio encoders, nor a systematic understanding of how pretraining objectives behave across diverse tasks and scales. We identify three key barriers: limited scale of audio-text corpora, limited coverage of audio attributes in existing caption corpora, and lack of systematic exploration and evaluation. To fill this gap, we present the first principled empirical study of ALP. We first introduce CaptionStew, a 10.7M caption dataset aggregating open-source audio-text corpora across multiple domains and captioning focuses. We then conduct the first comprehensive evaluation comparing contrastive and captioning objectives for learning audio representation across speech, music, and environmental sound tasks. Our results not only demonstrate that ALP yields competitive, transferable representations, but reveal critical trade-offs: contrastive learning offers superior data efficiency, while captioning exhibits better scalability. Furthermore, we find that the benefits of supervised initialization often diminish at larger scales, challenging common practices. By grounding these claims in empirical evidence, we establish a viable pathway toward general-purpose audio representation learning, guiding future research.
Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3
University of Texas at Austin · Tencent AI Lab Seattle · Zhejiang University +1
Contrastively pretrained audio-language models (e.g., CLAP) excel at clip-level understanding but struggle with frame-level tasks. Existing extensions fail to exploit the varying granularity of real-world audio-text data, where massive clip-level textual descriptions coexist with limited frame-level annotations. This paper proposes Fine-grained Language-Audio Pretraining (FineLAP), a novel training paradigm that advances both clip- and frame-level alignment in CLAP with heterogeneous data. FineLAP introduces a dual-stream sigmoid loss with a cluster-based sampling strategy to jointly learn from clip- and frame-level supervision. To capture both global semantics and local details, FineLAP uses a decoupled audio projector on top of a self-supervised encoder. To alleviate the scarcity of temporally annotated data, we present FineLAP-100k, a large-scale synthetic SED dataset constructed through a scalable curation pipeline. Extensive experiments demonstrate that FineLAP achieves SOTA performance across multiple audio understanding tasks, including retrieval, classification, sound event detection, and text-to-audio grounding. Ablation studies further show that coarse- and fine-grained alignment are mutually beneficial, providing insights for building better audio-language models (ALMs).
Xiquan Li, Xuenan Xu, Ziyang Ma +4
X-LANCE Lab, Shanghai Jiao Tong University, China · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China +1
Large audio-language models (LALMs) frequently hallucinate by overriding acoustic evidence with language priors. While contrastive decoding (CD) offers training-free mitigation, existing methods rely on blunt perturbations like masking or noise, leaving structured audio transformations unexplored. We explore this design space by evaluating a diverse library of targeted audio perturbations and adaptively selecting the optimal negative branch for each task and example. First, we improve upon earlier prompt engineering by showing that a simple binary yes/no constraint reduces the model's tendency to falsely confirm absent audio features. Second, evaluating our library across temporal, spectral, frequency, and amplitude domains reveals that optimal transformations are highly task-dependent; for instance, reversing the audio array disrupts temporal coherence, raising accuracy on the temporal order task from 74.7% to 81.4%. Finally, we trained a light-weight perturbation selector on model hidden states to dynamically route negative branches, yielding an additional +4.3% gain on the existence task.
Aaron Isidore Grace, Zhouyuan Huo, Weiran Wang
Department of Computer Science University of Iowa Iowa City, IA, USA · Google Mountain View, CA, USA