General audio understanding is a fundamental goal for large audio-language models, with audio captioning serving as a cornerstone task for their development. However, progress in this domain is hindered by existing datasets, which lack the scale and descriptive granularity required to train truly versatile models. To address this gap, we introduce ACAVCaps, a new large-scale, fine-grained, and multi-faceted audio captioning dataset. Derived from the ACAV100M collection, ACAVCaps is constructed using a multi-expert pipeline that analyzes audio from diverse perspectives-including speech, music, and acoustic properties-which are then synthesized into rich, detailed descriptions by a large language model. Experimental results demonstrate that models pre-trained on ACAVCaps exhibit substantially stronger generalization capabilities on various downstream tasks compared to those trained on other leading captioning datasets. The dataset is available at https://github.com/xiaomi-research/acavcaps.
Figures & tables
Figure 1: Data construction and evaluation frameowrk.
Labeling
Dataset
Duration (h)
Samples
Unique Tokens
Domain
Source
Manual
AudioCaps [ 15 ]
135
50k
5.5k
Multi-Domain ‡ , ⋄
AudioSet
Clotho [ 9 ]
24
3.8k
5.5k
Multi-Domain ‡ , ⋄
FreeSound
SongDescriber [ 17 ]
12
0.4k
2.4k
Music
MTG-Jamendo
LLM
MusicCaps [ 1 ]
7
4.6k
4.6k
Music
AudioSet
LPMusicCaps [ 8 ]
127
21.6k
5.1k
Music
Audioset, MSD
WavCaps [ 19 ]
1.8k
0.4M
23.1k
Multi-Domain ‡ , ⋄
AudioSet
Table 1: Comparison of ACAVCaps with existed caption datasets. The Unique Tokens column reports the total number of unique tokens within each dataset, as counted by the Qwen3 tokenizer. † MP-LLM: Multiple Experts Models and LLM; ‡ Multi-Domain: This includes speech, music and sound-events ( ⋄ denotes that domain were not elaborated in detail); \lx@sectionsign Extended Multi-Domain: This includes speech, music, sound-events, combinations thereof, and silence.
Training Dataset
Systematic
Content-Related
Content-Unrelated
Score
Long
Short
Speech
Music
Sound
Environment
Pure
Mixed
Pure
Mixed
Pure
Mixed
AudioSetCaps [ 3 ]
52.4
52.0
30.2
31.4
44.3
30.9
52.4
21.6
15.4
37.4
Auto-ACD [ 22 ]
47.3
50.0
29.1
31.0
26.9
21.9
49.5
18.9
11.0
32.8
WavCaps [ 19 ]
47.3
50.9
27.3
30.1
15.9
19.4
46.5
20.0
9.2
31.4
Sound-VeCaps [ 24 ]
47.0
49.7
29.1
30.3
27.2
21.9
49.8
18.7
11.4
32.8
Table 2: Performance of audio captioning models pre-trained on various datasets, evaluated on the MECAT-Caption benchmark. The final score is a weighted average of three main categories: systematic, Content-Specific, and Content-Unrelated. For all metrics, a higher score indicates better performance. Notably, a ’Pure’ sample contains only one content type (e.g., only speech), while a ’Mixed’ sample contains a combination of two or three types (i.e., speech, music, and sound events). † Combined refers to the combination of AudioSetCaps, Auto-ACD, WavCaps, and Sound-VECaps.
Training Dataset
Speech ↓
Sound ↑
Music ↑
Other ↑
AISHELL-2
LibriSpeech
Common Voice
General
Vocal
Instrument
Emotion
Android
IOS
MIC
Clean
Other
French
VGGSound
VocalSound
NSynth
IEMOCAP
AudioSetCaps
82.7
77.8
81.7
51.6
70.2
84.7
22.4
91.4
67.0
17.6
Auto-ACD
89.1
78.2
88.6
54.6
76.5
85.7
22.5
90.2
46.1
24.1
WavCaps
83.2
74.2
77.9
54.3
74.0
85.2
21.2
91.5
69.1
19.9
Sound-VECaps
87.3
79.5
87.9
51.8
70.1
85.6
22.9
90.8
45.0
20.3
Table 3: Performance on downstream tasks. A model is pretrained on each respective training dataset. Then, for each task, we freeze the model and only optimize the adapter. For all speech tasks, lower is better, while for all other tasks higher is better. † Combined refers to the combination of AudioSetCaps, Auto-ACD, WavCaps, and Sound-VECaps.