Authors: Adnan El Assadi, Isaac Chung, Chenghao Xiao, Roman Solomatin, Animesh Jha, Rahul Chand, Silky Singh, Kaitlyn Wang, Ali Sartaz Khan, Marc Moussa Nasser, Sufen Fong, Pengfei He, Alan Xiao, Ayush Sunil Munot, Aditya Shrivastava, Artem Gazizov, Niklas Muennighoff, Kenneth Enevoldsen
Organizations: Carleton University · Zendesk · Durham University · MIRAI · SaluteDevices · Stanford University · Indian Institute of Technology, Kharagpur · Capital One · Harvard University · Aarhus University
We introduce the Massive Audio Embedding Benchmark (MAEB), a large-scale benchmark covering 30 tasks across speech, music, environmental sounds, and cross-modal audio-text reasoning in 100+ languages. We evaluate 50+ models and find that no single model dominates across all tasks: contrastive audio-text models excel at environmental sound classification (e.g., ESC50) but score near random on multilingual speech tasks (e.g., SIB-FLEURS), while speech-pretrained models show the opposite pattern. Clustering remains challenging for all models, with even the best-performing model achieving only modest results. We observe that models excelling on acoustic understanding often perform poorly on linguistic tasks, and vice versa. We also show that the performance of audio encoders on MAEB correlates highly with their performance when used in audio large language models. MAEB is derived from MAEB+, a collection of 98 tasks. MAEB is designed to maintain task diversity while reducing evaluation cost, and it integrates into the MTEB ecosystem for unified evaluation across text, image, and audio modalities. We release MAEB and all 98 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.
Figures & tables
Figure 1: overview of task types and example subtypes in MAEB+. Values in parentheses denote numbers for MAEB.
Model
Params
MAEB
Extended
Speedup
YAMNet
3.7M
2.01
6.02
3.0 ×
wav2vec2-xls-r-2b
2B
26.93
45.62
1.7 ×
larger_clap_general
630M
11.52
32.23
2.8 ×
CLAP-htsat-fused
194M
13.03
35.35
2.7 ×
Table 1: Benchmark runtime comparison (GPU hours) between MAEB and MAEB(extended). Runtime measured on a single NVIDIA A100 GPU.
Figure 2: Domain-level performance on 94 tasks in MAEB+. Radial plot shows the top-performing model for each of the five acoustic domains: Speech (44 tasks), Music (13), Environmental (29), Bioacoustics (2), and Emotion (6). The dashed line represents an 80 target for universal performance, which remains unmet. Scores are averaged across all available task types (classification, clustering, retrieval, reranking). See Appendix D for methodology.
Rank ( ↓ )
Average
Average per Category
Model
MAEB
Audio
All
Cat.
Eng.
Multi.
Aud.
Clf
M.Clf
PC
Rrnk
Clust
A. Rtrvl
X. Rtrvl
Zero Clf.
MAEB
Number of datasets
(30)
(30)
(15)
(23)
(19)
(10)
(2)
(3)
(1)
(3)
(1)
(8)
(2)
Large audio-language models
LCO-Embedding-Omni-7B
1
5
52.2
55.6
50.9
53.6
52.2
58.0
45.7
67.3
78.7
1.7
78.2
50.3
64.5
Qwen2-Audio-7B
2
1
33.7
34.0
30.1
27.6
50.8
62.7
10.7
56.9
80.8
12.7
33.9
1.6
12.4
Table 2: Top 30 models on the MAEB benchmark (30 tasks spanning audio-only and audio-text evaluation). Results are ranked using Borda count. The “Audio” column shows the model’s rank on MAEB(audio-only) for reference. We provide averages across all tasks, and per task category. “Eng.” shows the average for English-only tasks, “Multi.” shows the average excluding tasks with no linguistic content (zxx), and “Aud.” shows the average for audio-only tasks. Task categories are abbreviated as: Classification (Clf), Multilabel Classification (M.Clf), Pair Classification (PC), Reranking (Rrnk), Clustering (Clust), Audio Retrieval (A. Rtrvl), Cross-modal Retrieval (X. Rtrvl), Zero-shot Classification (Zero Clf.). We highlight the best score in bold and the best score with each model category using a grey cell.
Figure 3: MAEB+ embedding quality correlates with Audio LLM performance. MMAU evaluates Audio LLMs across Speech, Music, and Sound, the same domains covered by MAEB+. Each point plots an Audio LLM’s overall MMAU score (y-axis, averaged across domains) against its encoder’s MAEB+ score (x-axis, computed from 26 classification tasks aligned with MMAU domains). Preliminary correlation (R²=0.86, p=0.072, n=4) suggests a positive relationship between embedding quality and downstream reasoning, though the small sample size and statistical marginality warrant caution in interpreting this relationship.
Figure 4: Language distribution in the MAEB+ collection. English dominates with 70 tasks. We use zxx (No Linguistic Content) to tag datasets with no languages present.
Dataset
Citation
MAEB
N. Samples
Total Duration(s)
N. Langs
Domains
Main Metric
Any2AnyRetrieval
JamAltArtistA2ARetrieval
( Cífka et al., 2024 )
✓
6.7k
22992
4
Music
ndcg_at_10
Classification
AmbientAcousticContext
( Park et al., 2020 )
1k
1046
1
Spoken, Speech
accuracy
BeijingOpera
( Tian et al., 2014 )
✓
236
393
1
Music
accuracy
BirdCLEF
( Klinck et al., 2025 )
✓
1k
33602
1
Spoken, Speech, Bioacoustics
accuracy
Table 3: MAEB+ Audio-Only Tasks Overview. Tasks are grouped by type and show MAEB benchmark membership, dataset size, total audio duration, language coverage, domains, and main evaluation metric. * denotes values from huge datasets.
Dataset
Citation
MAEB
N. Samples
Total Secs
N. Langs
Modality
Domains
Main Metric
Audio-to-Text Retrieval
AudioCapsA2TRetrieval
( Kim et al., 2019 )
5.3k
8708
2
a2t
Encyclopaedic, Written
cv_recall_at_5
AudioSetStrongA2TRetrieval
( Hershey et al., 2021 )
1k
5065
1
a2t
AudioScene
cv_recall_at_5
CMUArcticA2TRetrieval
( Clark and Richmond, 2003 )
2.6k
4134
1
a2t
Spoken
cv_recall_at_5
ClothoA2TRetrieval
( Drossos et al., 2019 )
6.6k
23636
1
a2t
Encyclopaedic, Written
cv_recall_at_5
CommonVoiceMini17A2TRetrieval
( Ardila et al., 2020 )
46.8k
120220
50
a2t
Spoken
cv_recall_at_5
Table 4: MAEB+ Audio-Text Cross-Modal Tasks Overview. Tasks include zero-shot classification and bidirectional retrieval between audio and text modalities, with dataset size, total audio duration, and main evaluation metric. * denotes values from huge datasets.
Appendix figures & tables50 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Domain distributions in the MAEB+ collection, MAEB, and MAEB(audio-only).
Model Name
Model Size
Modalities
laion/clap-htsat-fused ( Wu et al., 2024 )
153
audio, text
laion/clap-htsat-unfused ( Wu et al., 2024 )
153
audio, text
laion/larger_clap_general ( Wu et al., 2024 )
193
audio, text
laion/larger_clap_music ( Wu et al., 2024 )
193
audio, text
laion/larger_clap_music_and_speech ( Wu et al., 2024 )
193
audio, text
MIT/ast-finetuned-audioset-10-10-0.4593 ( Gong et al., 2021 )
86
audio
Appendix
Table 5: List of all models evaluated in MAEB. Model sizes are in millions of parameters.
Model
AmbientAcoustic
BeijingOpera
BirdCLEF
CommonLangAge
CommonLangGender
CREMA-D
ESC50
FSDD
Qwen/Qwen2-Audio-7B
45.33
97.45
37.10
17.59
48.83
73.99
96.30
90.33
LCO-Embedding/LCO-Embedding-Omni-7B
39.67
92.79
34.10
16.16
30.70
36.05
94.15
99.00
LCO-Embedding/LCO-Embedding-Omni-3B
38.42
93.22
31.60
16.87
33.22
31.03
94.40
98.27
openai/whisper-medium
39.85
91.54
29.00
17.63
36.63
53.98
84.00
87.73
openai/whisper-small
36.89
88.13
26.40
15.85
45.87
49.19
77.35
91.47
MIT/ast-finetuned-audioset-10-10-0.4593
48.86
97.03
45.20
12.82
52.33
37.84
96.20
64.27
Appendix
Table 6: English classification results (datasets 1–8 of 23).
Model
GTZANGenre
GunshotTri
IEMOCAPEmo
IEMOCAPGender
LibriCount
MridinghamStroke
MridinghamTonic
NSynth
Qwen/Qwen2-Audio-7B
93.10
100.00
29.96
92.96
49.60
84.33
61.17
63.04
LCO-Embedding/LCO-Embedding-Omni-7B
82.30
96.67
24.35
70.75
33.22
61.89
42.07
58.09
LCO-Embedding/LCO-Embedding-Omni-3B
81.00
96.60
22.53
59.42
30.35
61.52
39.54
59.02
openai/whisper-medium
76.00
94.25
25.60
76.05
57.87
69.21
49.51
51.33
openai/whisper-small
71.50
94.44
24.21
69.93
53.37
69.06
45.49
49.88
MIT/ast-finetuned-audioset-10-10-0.4593
80.70
98.82
20.49
87.00
42.15
79.20
54.16
56.26
Appendix
Table 7: English classification results (datasets 9–16 of 23).
Model
SpeechCommands
SpokenQA
TUTAcoustic
VocalSound
VoxCelebSA
VoxPopuliAccent
MInDS14
Qwen/Qwen2-Audio-7B
75.60
21.35
34.30
91.82
29.54
39.35
25.51
LCO-Embedding/LCO-Embedding-Omni-7B
94.21
36.58
25.35
91.77
43.40
10.33
98.14
LCO-Embedding/LCO-Embedding-Omni-3B
94.40
37.00
25.00
91.31
48.97
8.97
98.48
openai/whisper-medium
72.43
21.74
26.55
80.13
33.92
54.04
48.30
openai/whisper-small
73.80
19.96
23.55
77.59
32.45
38.85
35.64
MIT/ast-finetuned-audioset-10-10-0.4593
24.27
15.86
30.45
82.11
28.33
23.61
7.94
Appendix
Table 8: English classification results (datasets 17–23 of 23).
Model
cs
de
en
es
fr
it
ko
nl
pl
pt
ru
zh
LCO-Embedding/LCO-Embedding-Omni-7B
72.3
91.5
98.1
97.9
97.4
84.2
92.9
94.2
68.7
82.5
95.6
97.8
LCO-Embedding/LCO-Embedding-Omni-3B
72.5
89.8
98.5
96.9
97.4
84.8
93.2
92.1
67.6
81.0
95.2
98.2
facebook/seamless-m4t-v2-large
92.3
89.4
89.2
92.0
90.4
78.9
90.5
90.2
67.8
63.4
90.4
93.0
facebook/mms-1b-fl102
77.0
78.4
77.0
76.8
75.3
71.8
70.3
78.7
69.8
66.2
66.6
71.3
facebook/mms-1b-all
64.3
54.7
58.5
46.1
62.0
51.1
69.9
64.2
47.3
53.5
54.0
76.9
facebook/mms-1b-l1107
55.6
57.4
64.0
58.2
63.1
51.9
44.8
56.7
43.8
46.5
52.9
37.6
Appendix
Table 9: MInDS-14 classification results across languages. Best result per language in bold.
Model
afr_Latn
amh_Ethi
arb_Arab
asm_Beng
ast_Latn
azj_Latn
bel_Cyrl
ben_Beng
bos_Latn
bul_Cyrl
cat_Latn
ceb_Latn
ces_Latn
ckb_Arab
cym_Latn
LCO-Embedding/LCO-Embedding-Omni-7B
47.3
39.3
71.5
32.1
70.6
50.9
70.5
41.1
51.7
55.4
74.3
53.4
45.5
28.5
32.0
LCO-Embedding/LCO-Embedding-Omni-3B
40.2
35.7
65.2
36.6
65.4
53.6
63.3
32.9
51.8
49.2
68.8
48.1
53.6
21.4
21.4
facebook/seamless-m4t-v2-large
42.0
41.9
54.6
51.7
51.8
41.0
43.8
46.5
48.1
53.6
45.7
24.2
46.6
32.9
34.8
openai/whisper-medium
26.8
18.8
28.6
15.9
44.5
36.5
39.1
24.0
29.5
29.6
32.1
31.1
30.4
11.6
21.4
facebook/mms-1b-fl102
17.9
23.1
30.4
34.9
27.7
20.5
34.0
27.7
22.3
16.1
26.8
19.6
27.6
23.2
25.8
facebook/mms-1b-all
25.0
27.7
21.5
26.0
27.7
17.8
29.4
25.9
26.9
21.5
18.7
21.4
18.7
20.4
24.9
Appendix
Table 10: SIB-FLEURS classification results (languages 1–15 of 102). Best per language in bold.
Model
dan_Latn
deu_Latn
ell_Grek
eng_Latn
est_Latn
fin_Latn
fra_Latn
fuv_Latn
gaz_Latn
gle_Latn
glg_Latn
guj_Gujr
hau_Latn
heb_Hebr
hin_Deva
LCO-Embedding/LCO-Embedding-Omni-7B
39.3
71.3
29.4
70.6
34.8
40.2
81.3
25.8
39.4
28.6
72.2
64.3
22.4
28.5
73.2
LCO-Embedding/LCO-Embedding-Omni-3B
43.8
68.7
42.0
68.9
36.6
32.1
74.1
18.7
31.2
27.7
67.6
49.1
19.6
31.2
65.2
facebook/seamless-m4t-v2-large
49.1
52.7
38.4
57.2
49.1
47.2
57.3
15.2
11.5
33.0
53.6
49.8
16.0
43.6
46.5
openai/whisper-medium
31.3
41.0
27.7
28.6
28.5
28.6
34.8
19.6
16.1
20.4
22.3
26.0
17.8
15.9
31.3
facebook/mms-1b-fl102
25.8
29.6
17.7
20.6
27.7
21.3
20.6
24.2
30.3
32.2
26.8
25.8
25.9
25.8
25.1
facebook/mms-1b-all
17.0
33.2
18.7
21.5
19.7
26.8
27.6
16.1
19.7
23.2
22.3
26.8
21.5
19.6
28.7
Appendix
Table 11: SIB-FLEURS classification results (languages 16–30 of 102). Best per language in bold.
Model
hrv_Latn
hun_Latn
hye_Armn
ibo_Latn
ind_Latn
isl_Latn
ita_Latn
jav_Latn
jpn_Jpan
kam_Latn
kan_Knda
kat_Geor
kaz_Cyrl
kea_Latn
khk_Cyrl
LCO-Embedding/LCO-Embedding-Omni-7B
51.8
44.6
25.9
25.0
82.9
33.0
72.3
50.2
74.2
26.0
51.0
34.9
44.7
72.4
20.5
LCO-Embedding/LCO-Embedding-Omni-3B
55.4
29.4
29.4
22.3
78.4
24.0
67.9
43.8
72.3
29.5
44.7
21.4
41.0
65.2
24.2
facebook/seamless-m4t-v2-large
56.2
40.1
48.4
20.4
54.4
61.6
49.1
44.7
49.4
14.3
49.1
57.2
48.1
36.6
34.7
openai/whisper-medium
31.2
37.4
20.5
19.6
33.8
15.1
40.9
19.6
26.8
24.1
17.8
16.1
24.0
28.5
23.1
facebook/mms-1b-fl102
28.5
27.7
22.3
27.6
30.3
29.5
25.9
19.6
19.8
24.2
20.5
23.1
26.8
32.1
31.3
facebook/mms-1b-all
27.7
26.8
23.1
22.3
27.6
25.9
31.1
22.4
19.8
18.8
23.3
24.9
24.1
25.9
18.8
Appendix
Table 12: SIB-FLEURS classification results (languages 31–45 of 102). Best per language in bold.
Model
khm_Khmr
kir_Cyrl
kor_Hang
lao_Laoo
lin_Latn
lit_Latn
ltz_Latn
lug_Latn
luo_Latn
lvs_Latn
mal_Mlym
mar_Deva
mkd_Cyrl
mlt_Latn
mri_Latn
LCO-Embedding/LCO-Embedding-Omni-7B
23.3
43.8
68.7
49.1
32.9
42.7
65.2
21.4
36.6
39.2
38.3
56.2
49.8
59.7
21.4
LCO-Embedding/LCO-Embedding-Omni-3B
22.3
48.1
70.5
42.7
33.0
39.1
59.9
24.2
29.4
41.9
33.0
52.8
58.9
56.2
23.3
facebook/seamless-m4t-v2-large
30.4
55.1
40.4
33.2
23.2
55.3
37.5
35.7
21.4
52.8
44.5
41.9
64.3
43.8
25.9
openai/whisper-medium
26.8
20.4
27.7
24.0
30.4
32.8
32.0
20.6
19.5
34.9
25.0
34.7
32.2
21.3
28.6
facebook/mms-1b-fl102
27.7
24.9
19.7
30.4
26.7
32.0
26.0
27.7
25.7
35.6
21.3
27.6
25.1
24.9
24.2
facebook/mms-1b-all
18.7
16.1
29.6
23.2
23.1
23.3
22.3
25.0
22.3
26.6
25.0
32.9
26.1
28.7
22.4
Appendix
Table 13: SIB-FLEURS classification results (languages 46–60 of 102). Best per language in bold.
Model
mya_Mymr
nld_Latn
nob_Latn
npi_Deva
nso_Latn
nya_Latn
oci_Latn
ory_Orya
pan_Guru
pbt_Arab
pes_Arab
pol_Latn
por_Latn
ron_Latn
rus_Cyrl
LCO-Embedding/LCO-Embedding-Omni-7B
15.3
71.5
42.0
55.3
23.3
23.1
56.3
49.0
53.7
29.3
38.5
66.2
71.4
58.0
69.7
LCO-Embedding/LCO-Embedding-Omni-3B
20.5
66.0
50.9
46.5
23.3
26.8
55.2
37.7
41.1
29.4
27.5
71.3
75.9
57.2
75.0
facebook/seamless-m4t-v2-large
34.0
52.9
46.5
42.0
20.5
45.6
30.4
40.2
50.0
25.9
55.3
43.8
50.9
44.9
52.7
openai/whisper-medium
8.1
31.3
38.3
26.7
17.7
17.8
26.7
24.9
33.0
21.4
24.8
36.6
42.8
33.2
33.0
facebook/mms-1b-fl102
27.7
32.2
30.4
24.2
30.4
33.0
26.8
26.8
25.0
25.0
21.3
22.4
33.8
18.0
29.5
facebook/mms-1b-all
25.0
30.4
27.6
20.6
26.8
25.0
29.4
15.2
27.6
20.6
24.8
30.3
30.3
18.8
21.5
Appendix
Table 14: SIB-FLEURS classification results (languages 61–75 of 102). Best per language in bold.
Model
slk_Latn
slv_Latn
sna_Latn
snd_Arab
som_Latn
spa_Latn
srp_Cyrl
swe_Latn
swh_Latn
tam_Taml
tel_Telu
tgk_Cyrl
tgl_Latn
tha_Thai
tur_Latn
LCO-Embedding/LCO-Embedding-Omni-7B
51.8
55.3
32.3
38.4
22.2
75.1
59.8
50.9
29.5
24.0
50.8
34.7
50.2
72.2
43.9
LCO-Embedding/LCO-Embedding-Omni-3B
49.0
44.5
40.2
37.5
25.8
69.6
51.8
52.7
29.5
31.3
50.0
28.5
43.0
65.8
49.1
facebook/seamless-m4t-v2-large
50.1
53.6
24.2
20.6
30.3
53.4
56.2
45.6
41.8
41.1
48.2
42.6
37.7
44.5
49.1
openai/whisper-medium
33.1
29.3
28.6
23.2
15.2
30.3
33.0
27.7
26.8
27.7
29.4
28.5
26.6
34.7
33.0
facebook/mms-1b-fl102
37.5
30.3
24.2
26.6
21.4
19.6
23.2
22.2
37.5
26.7
23.2
26.0
20.6
32.8
28.7
facebook/mms-1b-all
22.3
24.1
27.8
25.8
28.6
21.4
25.0
18.7
33.1
24.0
24.9
24.1
15.1
24.0
24.3
Appendix
Table 15: SIB-FLEURS classification results (languages 76–90 of 102). Best per language in bold.
Model
ukr_Cyrl
umb_Latn
urd_Arab
uzn_Latn
vie_Latn
wol_Latn
xho_Latn
yor_Latn
zho_Hans
zho_Hant
zsm_Latn
zul_Latn
LCO-Embedding/LCO-Embedding-Omni-7B
68.6
17.8
55.5
51.1
69.6
34.0
31.1
27.8
67.7
64.2
63.4
26.0
LCO-Embedding/LCO-Embedding-Omni-3B
71.3
20.6
57.2
40.2
66.0
33.0
25.0
32.1
65.9
62.5
55.5
26.7
facebook/seamless-m4t-v2-large
54.4
15.3
43.8
58.9
50.8
25.1
19.7
33.8
49.0
33.1
51.8
16.1
openai/whisper-medium
32.1
17.8
21.5
23.0
30.4
14.2
15.2
16.9
22.3
29.5
30.5
16.0
facebook/mms-1b-fl102
32.0
27.7
24.9
31.4
22.4
28.3
22.4
26.7
26.0
24.2
26.7
26.8
facebook/mms-1b-all
27.8
25.8
23.1
31.2
22.4
18.7
24.0
29.4
22.3
19.8
27.7
28.7
Appendix
Table 16: SIB-FLEURS classification results (languages 91–102 of 102). Best per language in bold.
Model
GenderID
LanguageID
laion/larger_clap_general
84.6
84.6
Qwen/Qwen2-Audio-7B
68.2
99.0
laion/larger_clap_music_and_speech
84.4
81.4
openai/whisper-tiny
66.6
96.6
openai/whisper-small
62.8
99.2
openai/whisper-base
63.6
98.0
Appendix
Table 17: VoxPopuli classification results. GenderID = gender classification, LanguageID = language identification. Both tasks are evaluated on multilingual audio samples containing English, French, Spanish, Polish, and German. Best result per task in bold.
Model
AmbientAcoustic
CREMA-D
ESC50
GTZANGenre
MusicGenre
VehicleSound
VoiceGender
VoxCeleb
VoxPopuliAccent
laion/larger_clap_music_and_speech
43.51
14.43
94.46
63.82
45.30
4.72
71.33
0.70
10.61
laion/clap-htsat-unfused
43.68
12.78
94.76
55.70
40.07
2.66
68.11
1.06
9.35
laion/larger_clap_general
43.78
13.17
94.10
65.09
43.38
3.37
26.63
0.68
11.20
Qwen/Qwen2-Audio-7B
39.92
32.37
88.69
73.94
41.84
5.52
13.43
0.18
4.25
microsoft/msclap-2023
44.12
10.73
95.33
61.91
36.57
2.86
31.43
0.78
9.58
MIT/ast-finetuned-audioset-10-10-0.4593
46.76
7.20
91.36
61.60
47.20
13.37
14.07
1.01
7.55
Appendix
Table 18: English clustering results.
Model
VoxPopuliGender
laion/clap-htsat-fused
52.68
microsoft/msclap-2022
46.37
microsoft/msclap-2023
32.09
laion/clap-htsat-unfused
22.20
google/vggish
8.20
speechbrain/cnn14-esc50
7.72
Appendix
Table 19: VoxPopuli gender clustering results. Task clusters audio samples by speaker gender across multilingual audio samples containing German, English, French, Spanish, and Polish. Best result in bold.
Model
CREMA-D
ESC50
NMSQA
VocalSound
VoxPopuliAccent
LCO-Embedding/LCO-Embedding-Omni-7B
53.56
99.44
97.60
94.47
50.73
LCO-Embedding/LCO-Embedding-Omni-3B
52.40
99.54
96.92
94.65
50.67
microsoft/msclap-2023
57.65
99.22
51.09
84.22
52.05
laion/clap-htsat-unfused
56.91
99.31
46.98
81.58
53.96
laion/larger_clap_music_and_speech
56.66
99.35
46.14
80.36
53.49
laion/larger_clap_general
54.71
99.14
47.42
81.09
53.55
Appendix
Table 20: Pair classification results. Models are evaluated on audio pair similarity tasks. Best result per task in bold.
Model
AudioSetMini
BirdSet
FSD2019Kaggle
FSD50K
MIT/ast-finetuned-audioset-10-10-0.4593
55.10
6.55
36.90
2.79
LCO-Embedding/LCO-Embedding-Omni-7B
43.60
3.03
44.19
4.90
LCO-Embedding/LCO-Embedding-Omni-3B
42.75
3.05
42.97
4.22
google/yamnet
46.46
2.51
18.95
1.34
Qwen/Qwen2-Audio-7B
46.28
0.81
7.05
13.78
openai/whisper-medium
38.15
2.77
17.84
3.25
Appendix
Table 21: Multilabel classification results. Models are evaluated on audio tagging tasks where each sample can have multiple labels. Best result per task in bold.
Model
ESC50
Ravdess
SpeechCmd v0.01
SpeechCmd v0.02
UrbanSound8k
LCO-Embedding/LCO-Embedding-Omni-7B
87.80
31.67
96.96
97.42
67.38
LCO-Embedding/LCO-Embedding-Omni-3B
87.65
26.94
96.61
97.40
68.80
laion/larger_clap_general
90.50
17.29
12.39
12.44
79.64
microsoft/msclap-2023
89.85
15.21
9.78
10.01
83.01
laion/larger_clap_music_and_speech
82.85
17.29
9.23
9.03
76.86
microsoft/msclap-2022
80.25
13.61
10.09
10.51
76.17
Appendix
Table 22: Zero-shot classification results. Models classify audio using text descriptions without task-specific training. Best result per task in bold.
Model
ESC50
FSDnoisy18k
GTZAN
UrbanSound8K
VocalSound
LCO-Embedding/LCO-Embedding-Omni-7B
97.58
84.61
78.71
79.27
89.94
LCO-Embedding/LCO-Embedding-Omni-3B
97.16
83.19
75.37
77.93
89.28
microsoft/msclap-2023
97.98
85.34
75.43
76.50
72.12
Qwen/Qwen2-Audio-7B
94.34
68.44
80.85
70.66
80.29
MIT/ast-finetuned-audioset-10-10-0.4593
96.89
77.35
77.65
75.91
65.56
microsoft/msclap-2022
96.28
79.11
62.91
77.96
59.69
Appendix
Table 23: Reranking results. Models are evaluated on audio-to-audio reranking tasks. Best result per task in bold.
Model
AudioCaps A2T
AudioCaps T2A
AudioSetStrong A2T
AudioSetStrong T2A
CMU Arctic A2T
CMU Arctic T2A
Clotho A2T
Clotho T2A
EmoVDB A2T
LCO-Embedding/LCO-Embedding-Omni-7B
56.74
53.82
44.14
42.19
99.92
99.54
37.32
33.89
97.37
LCO-Embedding/LCO-Embedding-Omni-3B
54.25
46.95
41.21
34.77
99.70
99.70
34.74
32.70
98.34
laion/larger_clap_general
76.22
57.06
87.30
85.55
21.23
34.12
44.69
33.72
1.80
laion/larger_clap_music_and_speech
74.18
63.14
46.29
47.07
22.21
44.20
38.95
32.39
3.94
microsoft/msclap-2023
53.11
49.94
53.71
53.71
1.82
1.44
54.64
41.93
2.63
laion/clap-htsat-unfused
75.20
62.30
46.29
46.29
1.67
0.61
44.69
34.18
1.93
Appendix
Table 24: English retrieval results (datasets 1–9 of 27).
Model
EmoVDB T2A
GigaSpeech A2T
GigaSpeech T2A
HiFiTTS A2T
HiFiTTS T2A
JLCorpus A2T
JLCorpus T2A
LibriTTS A2T
LibriTTS T2A
LCO-Embedding/LCO-Embedding-Omni-7B
97.24
83.11
83.29
99.67
100.00
69.83
65.33
99.96
99.91
LCO-Embedding/LCO-Embedding-Omni-3B
98.69
82.96
83.13
99.67
100.00
70.50
70.67
99.96
99.94
laion/larger_clap_general
2.42
0.13
0.16
4.33
5.33
4.46
6.67
0.21
0.64
laion/larger_clap_music_and_speech
4.98
0.15
0.21
3.00
3.67
3.96
9.33
0.13
0.70
microsoft/msclap-2023
2.42
0.19
0.10
3.00
3.33
6.42
6.67
0.36
0.36
laion/clap-htsat-unfused
2.35
0.09
0.36
4.00
5.33
4.00
6.00
0.45
0.43
Appendix
Table 25: English retrieval results (datasets 10–18 of 27).
Model
MACS A2T
MACS T2A
MusicCaps A2T
MusicCaps T2A
SoundDescs A2T
SoundDescs T2A
SpokenSQuAD T2A
UrbanSound8K A2T
UrbanSound8K T2A
LCO-Embedding/LCO-Embedding-Omni-7B
16.03
29.77
20.93
24.83
21.63
32.79
74.00
0.79
0.92
LCO-Embedding/LCO-Embedding-Omni-3B
18.32
22.65
19.69
18.83
18.82
29.09
72.00
0.67
0.88
laion/larger_clap_general
30.03
33.08
17.36
16.49
23.45
22.96
2.00
0.94
0.96
laion/larger_clap_music_and_speech
28.24
30.53
13.36
14.02
24.07
24.44
2.00
1.02
0.94
microsoft/msclap-2023
15.52
27.23
18.48
19.22
38.14
37.28
0.00
0.90
0.94
laion/clap-htsat-unfused
25.19
27.99
11.70
11.89
20.52
21.33
1.33
0.98
0.98
Appendix
Table 26: English retrieval results (datasets 19–27 of 27).
Model
af
am
ar
as
ast
az
be
bg
bn
bs
ca
ceb
ckb
cmn
cs
cy
da
de
el
en
es
et
fa
ff
fi
fil
LCO-Embedding/LCO-Embedding-Omni-3B
88.3
41.9
98.8
43.1
100.0
82.2
98.3
91.2
59.8
87.4
99.9
80.8
47.6
100.0
89.3
35.5
65.3
100.0
61.8
100.0
100.0
57.9
37.3
62.0
50.4
80.5
LCO-Embedding/LCO-Embedding-Omni-7B
90.5
37.0
98.8
40.5
100.0
75.9
97.9
92.4
60.8
84.2
99.6
79.1
39.3
100.0
78.1
33.3
68.4
99.9
39.5
100.0
100.0
54.1
40.5
65.2
36.5
77.6
Qwen/Qwen2-Audio-7B
1.9
1.7
2.8
1.2
1.9
1.1
0.8
0.9
1.2
1.3
1.7
1.3
1.1
1.7
1.4
0.5
1.2
0.9
2.0
3.4
3.6
1.6
0.9
1.4
1.4
0.8
microsoft/speecht5_multimodal
2.3
1.0
2.1
1.1
0.4
0.7
0.6
1.7
0.5
0.4
1.1
1.3
1.1
1.0
1.7
1.1
1.4
0.9
1.1
1.4
0.7
0.7
0.5
1.1
1.5
0.6
laion/larger_clap_music_and_speech
2.7
1.4
1.4
0.2
1.9
0.8
0.4
1.5
0.5
0.9
0.7
1.7
0.5
0.5
1.0
0.8
1.2
0.3
0.6
0.9
1.2
0.6
0.6
1.1
0.5
0.3
laion/clap-htsat-fused
2.7
0.4
0.9
0.6
0.8
0.3
0.4
0.5
0.4
0.4
0.4
1.1
0.8
1.1
1.1
0.6
0.8
0.6
0.2
0.9
0.4
0.8
0.6
0.6
0.8
0.3
Appendix
Table 27: FLEURS A2T retrieval results (languages 1–26 of 102). Best per language in bold.
Model
fr
ga
gl
gu
ha
he
hi
hr
hu
hy
id
ig
is
it
ja
jv
ka
kam
kea
kk
km
kn
ko
ky
lb
lg
LCO-Embedding/LCO-Embedding-Omni-3B
100.0
29.2
100.0
91.8
40.4
38.3
99.0
95.1
35.5
36.6
99.9
35.5
73.9
100.0
100.0
89.3
52.6
33.9
98.0
61.4
16.0
64.0
100.0
75.3
92.7
44.5
LCO-Embedding/LCO-Embedding-Omni-7B
100.0
27.6
100.0
92.2
31.1
37.2
98.8
87.7
31.7
27.5
99.9
32.7
54.3
100.0
100.0
89.3
47.4
38.0
98.6
59.3
16.5
64.9
100.0
69.7
93.3
42.3
Qwen/Qwen2-Audio-7B
3.3
1.2
2.9
1.2
0.8
1.0
1.9
1.5
1.2
1.3
2.5
0.8
21.7
3.1
2.9
0.8
1.3
1.0
1.5
1.6
0.4
0.6
2.6
1.4
1.2
1.0
microsoft/speecht5_multimodal
0.9
1.1
1.1
1.1
1.1
1.0
2.2
0.3
0.9
0.9
1.6
0.7
15.2
1.2
0.9
1.1
0.9
1.0
0.6
0.6
0.6
0.7
1.3
0.7
1.2
0.7
laion/larger_clap_music_and_speech
0.9
0.8
0.9
0.3
1.0
1.1
0.7
0.7
0.7
0.4
1.2
0.5
10.9
1.0
1.1
0.5
0.4
1.0
0.8
0.8
0.4
0.7
2.1
0.4
0.6
1.0
laion/clap-htsat-fused
0.4
0.6
0.5
0.6
1.3
0.9
0.7
0.5
1.1
0.8
1.2
0.7
13.0
0.8
0.9
0.5
0.3
0.7
0.5
0.5
1.0
0.2
1.3
0.5
0.9
0.6
Appendix
Table 28: FLEURS A2T retrieval results (languages 27–52 of 102). Best per language in bold.
Model
ln
lo
lt
luo
lv
mi
mk
ml
mn
mr
ms
mt
my
nb
ne
nl
nso
ny
oc
om
or
pa
pl
ps
pt
ro
LCO-Embedding/LCO-Embedding-Omni-3B
43.1
54.3
51.8
66.0
57.9
23.8
95.3
55.8
31.6
80.6
99.6
89.5
6.2
92.2
80.0
100.0
43.4
43.5
91.7
97.6
64.9
85.0
98.3
59.8
100.0
96.5
LCO-Embedding/LCO-Embedding-Omni-7B
47.5
74.6
55.2
68.0
56.5
22.4
92.9
56.5
22.1
82.6
99.9
89.0
6.7
83.8
80.9
99.7
43.3
41.5
93.1
90.2
65.3
89.9
86.3
60.0
100.0
93.2
Qwen/Qwen2-Audio-7B
0.4
4.0
0.6
1.6
2.1
0.8
1.5
0.8
0.9
0.9
1.6
1.9
0.7
4.5
1.1
2.2
0.6
0.8
1.0
14.6
1.5
1.2
1.1
1.0
3.7
1.2
microsoft/speecht5_multimodal
1.7
1.7
0.5
3.1
1.4
0.7
0.8
0.7
0.6
0.9
1.3
0.8
0.6
3.1
0.6
1.9
1.0
1.2
0.7
12.2
0.9
1.2
0.5
2.0
1.0
1.0
laion/larger_clap_music_and_speech
1.0
1.7
0.9
3.1
0.7
1.2
0.7
0.5
0.6
0.1
0.8
0.5
0.6
2.5
0.7
2.7
1.1
0.5
0.7
19.5
0.5
0.3
1.2
1.6
0.8
0.8
laion/clap-htsat-fused
1.5
1.0
0.4
2.0
0.7
0.4
0.2
0.7
0.4
0.2
0.7
0.5
0.1
1.4
0.4
1.4
1.3
0.8
0.8
19.5
0.6
0.7
0.5
1.0
0.4
0.9
Appendix
Table 29: FLEURS A2T retrieval results (languages 53–78 of 102). Best per language in bold.
Model
ru
sd
sk
sl
sn
so
sr
sv
sw
ta
te
tg
th
tr
uk
umb
ur
uz
vi
wo
xh
yo
yue
zu
LCO-Embedding/LCO-Embedding-Omni-3B
100.0
77.3
96.6
78.1
45.1
37.8
89.7
86.6
44.1
32.8
85.8
66.5
99.7
99.1
98.3
35.1
99.0
52.9
99.2
66.8
43.1
29.5
100.0
33.0
LCO-Embedding/LCO-Embedding-Omni-7B
100.0
76.6
92.2
83.8
43.7
33.3
89.6
78.1
48.9
26.2
72.0
69.3
99.7
97.7
98.9
30.1
99.0
54.5
99.1
66.8
39.4
31.5
100.0
27.9
Qwen/Qwen2-Audio-7B
2.2
0.7
1.8
1.0
1.0
1.0
1.7
1.6
1.8
0.5
1.9
1.3
0.7
1.5
1.5
0.8
3.0
0.5
1.9
2.2
0.9
0.5
1.3
1.3
microsoft/speecht5_multimodal
1.3
0.9
0.6
0.5
0.8
1.2
1.0
1.3
1.8
1.4
1.5
1.2
0.6
0.7
1.2
1.1
2.0
1.3
0.6
1.6
0.7
1.1
1.5
0.7
laion/larger_clap_music_and_speech
0.8
0.9
1.3
1.6
0.8
0.5
0.9
1.1
0.8
1.0
1.3
1.5
0.3
0.5
0.7
0.5
2.0
0.5
1.2
1.6
0.8
0.5
0.6
0.8
laion/clap-htsat-fused
0.8
0.5
0.8
0.5
0.9
0.4
0.9
0.9
1.0
1.4
1.1
0.7
0.2
0.9
1.2
0.5
1.3
0.5
0.4
0.8
0.3
0.2
0.9
0.7
Appendix
Table 30: FLEURS A2T retrieval results (languages 79–102 of 102). Best per language in bold.
Model
af
am
ar
as
ast
az
be
bg
bn
bs
ca
ceb
ckb
cmn
cs
cy
da
de
el
en
es
et
fa
ff
fi
fil
LCO-Embedding/LCO-Embedding-Omni-3B
92.4
26.9
97.9
47.9
100.0
82.8
98.1
91.9
64.5
87.1
99.4
88.4
43.2
99.9
90.2
26.1
73.7
99.9
69.2
100.0
100.0
66.9
45.0
65.3
58.9
86.5
LCO-Embedding/LCO-Embedding-Omni-7B
90.2
25.6
98.6
42.5
99.8
76.4
97.5
93.0
62.9
82.2
98.7
81.9
38.0
100.0
79.5
26.3
68.5
99.9
44.5
100.0
100.0
55.2
43.7
54.7
39.8
80.8
Qwen/Qwen2-Audio-7B
3.4
2.3
1.4
0.6
2.5
0.5
1.2
1.5
0.8
1.0
1.3
1.3
1.0
1.7
1.0
0.4
0.5
1.2
1.7
4.3
2.4
0.9
0.9
1.2
0.5
0.4
microsoft/speecht5_multimodal
1.5
1.4
1.2
1.4
0.4
1.1
0.7
1.5
0.9
0.6
0.5
1.1
0.7
1.0
1.4
0.7
1.2
1.2
1.1
3.4
0.6
0.9
0.9
1.1
1.0
0.5
laion/larger_clap_music_and_speech
3.0
1.0
1.4
0.5
1.9
0.8
0.6
0.6
0.5
1.7
1.2
0.9
0.7
0.6
1.9
0.8
0.6
1.0
0.9
1.4
1.0
1.1
0.8
0.8
0.9
1.0
laion/larger_clap_general
4.2
1.0
0.7
0.5
1.5
0.7
0.4
0.8
0.5
1.1
0.5
0.9
0.4
0.4
1.2
0.5
1.0
1.3
0.6
1.9
0.8
0.8
0.6
1.8
0.9
0.8
Appendix
Table 31: FLEURS T2A retrieval results (languages 1–26 of 102). Best per language in bold.
Model
fr
ga
gl
gu
ha
he
hi
hr
hu
hy
id
ig
is
it
ja
jv
ka
kam
kea
kk
km
kn
ko
ky
lb
lg
LCO-Embedding/LCO-Embedding-Omni-3B
99.9
29.8
99.9
91.8
32.5
43.8
98.3
94.6
43.1
39.4
99.9
31.9
71.7
100.0
99.7
91.5
59.4
30.8
97.5
67.8
15.4
72.1
100.0
75.1
92.9
38.3
LCO-Embedding/LCO-Embedding-Omni-7B
100.0
27.2
100.0
92.1
31.4
42.8
98.6
87.3
36.2
32.2
99.9
23.3
76.1
100.0
99.8
93.4
52.8
33.6
97.9
59.8
15.7
73.5
100.0
70.4
92.2
36.9
Qwen/Qwen2-Audio-7B
2.5
0.5
2.3
1.3
0.5
1.3
1.2
1.1
0.4
0.8
1.0
0.5
19.6
2.7
0.9
0.8
0.8
0.6
0.7
0.7
0.9
1.0
2.4
0.5
0.6
1.1
microsoft/speecht5_multimodal
1.0
0.7
0.9
1.2
1.0
1.0
1.9
0.9
1.0
1.0
1.6
1.3
13.0
0.7
0.8
1.2
0.6
1.1
0.9
1.1
0.9
0.7
1.6
0.5
0.7
1.1
laion/larger_clap_music_and_speech
2.5
1.2
1.3
0.5
0.5
0.5
1.2
1.3
0.8
0.4
1.0
0.4
10.9
2.0
0.5
0.8
0.6
1.0
1.4
0.6
0.6
0.6
1.8
0.6
0.9
1.0
laion/larger_clap_general
1.9
1.4
0.4
0.4
0.6
0.6
1.4
1.1
1.2
0.5
0.9
0.8
17.4
1.7
1.2
1.4
0.4
1.0
0.8
0.6
0.6
0.1
1.3
0.1
1.0
1.5
Appendix
Table 32: FLEURS T2A retrieval results (languages 27–52 of 102). Best per language in bold.
Model
ln
lo
lt
luo
lv
mi
mk
ml
mn
mr
ms
mt
my
nb
ne
nl
nso
ny
oc
om
or
pa
pl
ps
pt
ro
LCO-Embedding/LCO-Embedding-Omni-3B
45.6
40.7
63.2
73.0
66.4
23.2
95.0
65.2
24.1
82.5
99.6
89.1
5.2
93.8
82.0
99.7
40.0
38.2
91.0
73.2
62.2
83.4
97.0
63.5
100.0
95.7
LCO-Embedding/LCO-Embedding-Omni-7B
43.7
64.0
58.2
65.2
58.6
21.0
92.2
64.2
21.3
80.5
99.6
86.5
4.3
84.9
80.2
99.7
31.9
35.5
90.4
70.7
62.5
87.3
84.2
59.0
100.0
91.4
Qwen/Qwen2-Audio-7B
1.0
2.0
0.6
2.3
0.9
0.8
0.3
1.1
0.7
0.3
1.2
0.5
0.9
3.1
1.0
1.4
0.5
0.1
0.5
17.1
1.5
1.0
0.9
1.2
2.5
1.2
microsoft/speecht5_multimodal
1.5
1.5
0.8
3.9
0.5
0.5
0.8
0.5
0.4
0.6
0.7
0.6
0.8
2.5
1.5
2.2
1.0
1.2
1.0
19.5
0.7
0.9
1.3
1.2
1.2
0.9
laion/larger_clap_music_and_speech
2.1
1.5
1.1
2.3
1.3
1.0
0.6
0.6
0.4
0.6
0.8
1.1
0.5
1.7
1.0
2.5
0.9
0.7
0.6
12.2
0.7
0.5
2.4
1.2
2.6
1.5
laion/larger_clap_general
1.9
1.5
1.3
3.1
0.5
1.0
0.5
0.3
0.3
0.3
0.8
1.3
0.7
2.0
0.4
2.2
1.0
1.1
0.7
7.3
0.3
0.5
1.2
0.8
1.3
1.4
Appendix
Table 33: FLEURS T2A retrieval results (languages 53–78 of 102). Best per language in bold.
Model
ru
sd
sk
sl
sn
so
sr
sv
sw
ta
te
tg
th
tr
uk
umb
ur
uz
vi
wo
xh
yo
yue
zu
LCO-Embedding/LCO-Embedding-Omni-3B
100.0
75.8
95.6
83.2
35.5
32.1
91.1
83.7
45.0
36.2
85.2
66.3
99.3
99.2
98.3
36.7
100.0
56.8
99.0
58.0
33.6
33.8
99.9
20.8
LCO-Embedding/LCO-Embedding-Omni-7B
100.0
74.8
91.5
83.2
32.3
21.6
91.0
75.4
47.4
33.3
74.8
68.0
99.7
98.0
97.5
29.8
99.0
53.8
99.2
56.1
28.8
31.9
100.0
19.3
Qwen/Qwen2-Audio-7B
1.4
0.4
0.9
1.0
0.5
0.7
1.3
0.5
2.1
0.8
1.7
1.2
0.7
1.1
1.6
1.6
2.0
0.5
1.3
1.9
0.8
1.1
0.7
0.8
microsoft/speecht5_multimodal
1.2
0.6
1.0
1.2
0.6
0.4
1.7
1.3
1.6
1.4
1.3
1.2
0.5
0.5
0.8
1.8
2.3
0.8
0.5
1.3
1.0
1.0
1.1
1.2
laion/larger_clap_music_and_speech
0.8
0.5
1.3
1.0
0.4
0.8
2.4
1.7
1.2
0.7
1.1
0.8
0.5
0.9
1.2
1.3
1.7
0.5
0.8
1.3
0.8
0.7
0.6
0.9
laion/larger_clap_general
0.8
0.5
1.6
0.8
0.8
0.5
2.0
1.3
0.8
1.0
0.8
0.8
0.4
1.8
0.8
2.1
1.3
0.8
0.9
1.6
0.8
0.5
0.7
1.1
Appendix
Table 34: FLEURS T2A retrieval results (languages 79–102 of 102). Best per language in bold.
Model
de
en
es
fr
LCO-Embedding/LCO-Embedding-Omni-3B
73.5
76.4
81.4
72.5
LCO-Embedding/LCO-Embedding-Omni-7B
66.1
71.1
77.2
71.8
microsoft/msclap-2023
0.9
0.7
1.8
1.0
Qwen/Qwen2-Audio-7B
0.7
0.8
1.1
1.5
microsoft/speecht5_multimodal
0.7
1.7
0.8
0.8
laion/larger_clap_music_and_speech
1.0
0.5
0.8
1.6
Appendix
Table 35: JamAlt A2T retrieval results (languages 1–4 of 4). Best per language in bold.
Model
de
en
es
fr
LCO-Embedding/LCO-Embedding-Omni-3B
75.6
79.6
81.7
75.4
LCO-Embedding/LCO-Embedding-Omni-7B
72.4
78.6
80.8
76.0
Qwen/Qwen2-Audio-7B
1.2
2.0
1.7
1.8
microsoft/speecht5_multimodal
1.1
3.0
1.0
0.9
laion/larger_clap_music_and_speech
1.5
1.3
0.6
1.0
laion/clap-htsat-unfused
0.8
1.5
0.9
0.7
Appendix
Table 36: JamAlt T2A retrieval results (languages 1–4 of 4). Best per language in bold.
Model
de
en
es
fr
laion/larger_clap_music_and_speech
99.1
94.3
97.9
96.3
OpenMuQ/MuQ-MuLan-large
97.6
95.2
97.6
94.7
laion/larger_clap_general
97.9
93.2
96.8
94.6
MIT/ast-finetuned-audioset-10-10-0.4593
97.7
90.2
96.2
94.4
laion/clap-htsat-unfused
97.0
88.8
94.2
91.8
microsoft/msclap-2023
96.5
87.3
94.9
92.5
Appendix
Table 37: JamAlt A2A retrieval results (languages 1–4 of 4). Best per language in bold.
Model
ar
ast
be
bg
bn
br
cs
cy
da
de
el
en
es
LCO-Embedding/LCO-Embedding-Omni-3B
94.4
98.3
99.4
69.4
58.0
67.4
86.6
38.4
56.6
100.0
46.4
99.8
100.0
LCO-Embedding/LCO-Embedding-Omni-7B
94.8
99.2
99.4
66.2
56.6
56.6
70.2
27.2
49.0
99.8
21.6
99.6
100.0
Qwen/Qwen2-Audio-7B
2.6
8.4
2.8
2.8
2.0
1.4
2.6
1.4
2.2
4.4
2.6
4.2
6.2
microsoft/speecht5_multimodal
1.2
8.4
0.2
1.0
0.8
1.6
1.8
1.0
1.6
1.2
0.6
1.0
2.0
laion/clap-htsat-unfused
1.0
3.4
1.4
1.4
1.6
2.0
2.2
1.2
1.2
1.2
1.4
1.4
0.4
microsoft/msclap-2023
0.8
4.2
1.4
1.0
0.8
0.8
1.8
1.2
1.6
0.8
1.6
1.2
1.6
Appendix
Table 38: CommonVoiceMini17 A2T retrieval results (languages 1–13 of 50). Best per language in bold.
Model
et
fa
fi
fr
frold
gl
ha
hi
hu
it
ja
ka
ko
LCO-Embedding/LCO-Embedding-Omni-3B
46.8
25.2
39.0
99.8
99.8
100.0
61.6
94.0
50.0
99.6
95.2
61.2
99.1
LCO-Embedding/LCO-Embedding-Omni-7B
39.4
21.4
24.2
99.4
99.4
100.0
51.2
94.4
42.4
99.6
95.8
54.4
98.8
Qwen/Qwen2-Audio-7B
2.6
2.8
2.0
4.6
4.6
3.6
2.4
1.4
2.0
6.0
3.4
2.8
4.4
microsoft/speecht5_multimodal
1.4
1.4
3.2
2.4
2.4
1.8
1.8
0.8
1.8
2.0
1.2
1.2
1.2
laion/clap-htsat-unfused
1.0
1.0
1.2
0.6
0.6
1.4
2.1
1.4
1.2
0.8
1.4
1.2
2.4
microsoft/msclap-2023
1.0
1.2
2.0
1.6
1.6
1.0
1.5
1.0
1.0
1.4
1.0
1.6
2.1
Appendix
Table 39: CommonVoiceMini17 A2T retrieval results (languages 14–26 of 50). Best per language in bold.
Model
lt
lv
mk
ml
mn
mr
nl
oc
pl
pt
ro
ru
sk
LCO-Embedding/LCO-Embedding-Omni-3B
62.8
43.4
76.2
57.8
20.2
78.6
99.0
90.9
88.4
99.0
86.0
99.2
82.2
LCO-Embedding/LCO-Embedding-Omni-7B
59.6
38.8
72.9
40.8
11.8
81.2
99.0
90.2
66.2
98.8
78.8
99.2
80.2
Qwen/Qwen2-Audio-7B
1.8
2.2
3.3
2.2
2.0
2.0
2.8
5.1
2.6
2.4
2.6
3.0
2.4
microsoft/speecht5_multimodal
2.4
2.0
1.5
0.8
1.0
2.4
2.2
2.0
1.8
1.6
1.0
0.6
1.8
laion/clap-htsat-unfused
0.8
1.4
1.5
1.6
1.2
1.4
0.8
2.4
1.8
0.6
1.0
1.6
1.0
microsoft/msclap-2023
1.4
0.4
1.5
1.4
0.8
1.2
1.0
2.4
1.8
1.2
1.4
1.4
0.8
Appendix
Table 40: CommonVoiceMini17 A2T retrieval results (languages 27–39 of 50). Best per language in bold.
Model
sl
sr
sv
sw
ta
te
th
tr
uk
ur
vi
LCO-Embedding/LCO-Embedding-Omni-3B
59.8
58.6
68.2
53.0
42.4
91.8
92.0
72.2
86.6
82.2
84.4
LCO-Embedding/LCO-Embedding-Omni-7B
63.0
54.0
46.8
49.4
21.6
59.2
92.0
63.4
86.6
84.6
86.6
Qwen/Qwen2-Audio-7B
1.2
1.8
2.2
2.0
2.0
16.3
1.4
1.2
2.2
1.2
1.6
microsoft/speecht5_multimodal
1.0
0.4
1.8
2.4
0.6
6.1
0.8
0.8
1.6
1.0
1.0
laion/clap-htsat-unfused
0.8
1.2
1.0
1.4
1.0
12.2
1.2
2.0
1.2
1.2
1.4
microsoft/msclap-2023
2.0
2.0
0.4
1.4
1.4
10.2
1.4
1.2
1.2
1.2
0.8
Appendix
Table 41: CommonVoiceMini17 A2T retrieval results (languages 40–50 of 50). Best per language in bold.
Model
ar
ast
be
bg
bn
br
cs
cy
da
de
el
en
es
LCO-Embedding/LCO-Embedding-Omni-3B
94.2
99.2
98.8
73.2
58.2
51.0
87.4
20.0
63.2
99.8
41.2
99.6
100.0
LCO-Embedding/LCO-Embedding-Omni-7B
95.0
97.5
99.0
65.6
49.4
47.0
72.6
16.2
56.0
99.6
19.4
99.8
100.0
Qwen/Qwen2-Audio-7B
1.6
8.4
3.0
1.0
2.2
1.8
2.0
2.8
2.2
3.0
1.0
5.2
5.6
microsoft/speecht5_multimodal
1.0
9.2
1.0
1.4
1.4
2.0
1.8
1.6
1.4
1.6
0.8
2.0
1.4
microsoft/msclap-2023
0.8
10.1
1.0
1.6
1.0
1.6
1.2
1.8
1.6
1.6
1.0
0.8
1.6
laion/clap-htsat-fused
0.6
4.2
1.6
1.8
0.6
1.2
1.6
1.6
1.6
1.8
1.2
1.2
0.8
Appendix
Table 42: CommonVoiceMini17 T2A retrieval results (languages 1–13 of 50). Best per language in bold.
Model
et
fa
fi
fr
frold
gl
ha
hi
hu
it
ja
ka
ko
LCO-Embedding/LCO-Embedding-Omni-3B
46.6
25.6
41.6
99.6
99.6
99.8
49.7
92.8
48.8
99.6
95.0
58.8
98.8
LCO-Embedding/LCO-Embedding-Omni-7B
41.8
26.0
22.0
99.4
99.4
99.8
40.9
91.8
42.2
99.8
96.2
50.6
98.5
Qwen/Qwen2-Audio-7B
2.8
1.4
0.8
4.2
4.2
4.4
2.4
1.6
1.2
6.4
1.4
2.4
4.4
microsoft/speecht5_multimodal
2.0
1.0
1.8
1.0
1.0
1.4
1.8
1.2
1.4
1.2
1.0
0.8
0.9
microsoft/msclap-2023
1.8
1.2
1.4
1.2
1.2
0.8
0.6
1.2
1.4
1.2
0.8
0.8
2.1
laion/clap-htsat-fused
1.2
1.0
1.8
1.4
1.4
1.2
1.5
1.0
1.2
0.4
1.2
1.0
1.8
Appendix
Table 43: CommonVoiceMini17 T2A retrieval results (languages 14–26 of 50). Best per language in bold.
Model
lt
lv
mk
ml
mn
mr
nl
oc
pl
pt
ro
ru
sk
LCO-Embedding/LCO-Embedding-Omni-3B
66.6
47.8
78.3
52.2
11.6
80.8
98.6
91.7
88.6
98.4
84.0
99.2
80.8
LCO-Embedding/LCO-Embedding-Omni-7B
55.6
40.4
75.0
32.4
6.6
77.0
98.4
90.2
66.4
98.0
73.8
99.0
79.0
Qwen/Qwen2-Audio-7B
1.8
1.4
3.6
2.8
1.6
1.4
2.8
6.3
1.6
3.4
2.8
2.2
2.8
microsoft/speecht5_multimodal
2.0
2.0
1.8
1.0
0.8
0.8
1.6
1.6
1.6
1.4
1.8
1.4
1.0
microsoft/msclap-2023
1.6
1.2
2.1
1.4
1.4
1.0
1.2
2.4
2.0
1.0
1.4
1.0
2.0
laion/clap-htsat-fused
1.4
2.0
1.8
0.8
1.4
0.8
1.2
2.8
1.4
2.0
0.8
1.8
2.4
Appendix
Table 44: CommonVoiceMini17 T2A retrieval results (languages 27–39 of 50). Best per language in bold.
Model
sl
sr
sv
sw
ta
te
th
tr
uk
ur
vi
LCO-Embedding/LCO-Embedding-Omni-3B
65.2
49.8
67.0
47.0
29.8
73.5
90.4
72.2
91.4
80.8
81.8
LCO-Embedding/LCO-Embedding-Omni-7B
66.4
44.6
50.8
41.2
23.2
61.2
90.8
63.6
86.6
83.4
84.6
Qwen/Qwen2-Audio-7B
1.4
1.0
2.4
1.2
0.8
14.3
1.6
1.2
2.2
2.4
1.6
microsoft/speecht5_multimodal
1.0
1.2
1.4
1.8
1.0
14.3
0.8
1.4
0.8
0.8
1.2
microsoft/msclap-2023
2.0
1.8
1.2
0.6
0.8
8.2
1.4
0.8
1.2
0.8
0.8
laion/clap-htsat-fused
0.6
0.2
1.2
1.4
1.6
10.2
1.4
1.8
0.8
1.0
1.8
Appendix
Table 45: CommonVoiceMini17 T2A retrieval results (languages 40–50 of 50). Best per language in bold.
Model
ab
af
am
ar
as
ast
az
ba
bas
be
bg
bn
br
ca
ckb
cnh
cs
cv
cy
da
dav
de
dv
dyu
el
en
eo
es
et
LCO-Embedding/LCO-Embedding-Omni-3B
53.5
85.5
55.0
92.5
41.5
99.4
83.7
52.0
74.5
99.0
83.0
71.5
79.0
98.0
65.5
66.0
83.5
76.5
45.5
63.0
63.5
99.5
14.0
93.7
51.5
98.5
91.0
100.0
52.0
LCO-Embedding/LCO-Embedding-Omni-7B
44.0
85.5
35.5
93.0
26.5
97.5
81.5
46.5
69.5
98.0
79.0
64.5
66.5
96.5
47.5
62.5
69.0
65.5
33.5
56.0
53.5
99.5
5.5
88.9
30.5
98.5
88.5
100.0
49.0
Qwen/Qwen2-Audio-7B
5.0
7.7
5.0
4.5
7.0
9.6
12.0
3.0
5.5
4.0
4.0
4.0
3.5
6.5
6.0
3.5
5.0
4.5
5.5
5.0
4.0
8.0
4.5
6.3
6.0
8.5
6.5
9.5
5.0
microsoft/speecht5_multimodal
2.5
6.0
3.0
3.0
3.5
5.7
7.6
3.0
3.0
3.0
2.0
4.5
2.0
4.0
2.0
4.5
3.0
2.5
5.0
7.0
2.0
4.0
1.5
9.5
1.5
6.0
4.5
2.5
4.0
laion/clap-htsat-unfused
2.5
5.1
1.5
3.5
2.0
3.2
4.3
4.5
4.0
1.5
3.5
2.5
6.0
4.5
3.0
3.5
3.5
1.5
3.5
3.0
4.0
3.5
2.5
12.7
3.0
4.0
3.5
3.0
2.5
laion/clap-htsat-fused
3.0
5.1
2.0
2.5
1.5
5.1
2.2
3.0
3.5
1.5
3.5
1.5
3.5
4.0
1.5
4.5
3.0
3.5
3.5
2.0
1.5
1.5
1.5
9.5
4.0
3.5
4.0
1.5
2.5
Appendix
Table 46: CommonVoiceMini21 A2T retrieval results (languages 1–29 of 114). Best per language in bold.
Model
eu
fa
fi
fr
fy
ga
gl
gn
ha
he
hi
hsb
hu
hy
ia
id
it
ja
ka
kab
kk
kln
kmr
ko
ky
lg
lij
lt
ltg
LCO-Embedding/LCO-Embedding-Omni-3B
75.0
38.5
53.5
100.0
93.0
41.0
100.0
76.0
69.5
40.0
97.0
72.5
56.0
49.5
99.5
96.5
100.0
93.0
72.0
52.5
51.0
68.0
75.0
99.5
61.0
57.5
95.5
61.2
67.0
LCO-Embedding/LCO-Embedding-Omni-7B
69.0
33.5
33.0
100.0
92.0
35.5
100.0
58.0
55.5
31.5
96.0
73.5
45.5
36.5
99.5
95.5
100.0
94.5
63.0
42.5
31.5
57.0
59.0
98.0
57.0
46.0
92.5
63.6
56.4
Qwen/Qwen2-Audio-7B
4.0
5.0
5.5
5.0
4.5
3.0
9.5
5.5
3.5
3.0
5.5
7.5
3.5
4.0
6.0
5.5
8.5
5.5
4.0
3.5
4.5
5.0
5.5
3.5
4.0
5.0
6.0
1.8
2.2
microsoft/speecht5_multimodal
2.0
2.0
7.0
5.5
6.0
3.5
4.0
4.0
5.5
3.0
2.5
4.5
4.0
3.5
4.5
5.0
4.5
3.0
2.5
2.5
3.0
3.5
4.5
1.5
2.5
3.5
5.0
2.4
1.8
laion/clap-htsat-unfused
4.5
1.5
3.0
3.0
3.0
2.5
2.0
4.0
4.0
2.0
1.0
4.5
3.5
2.5
4.0
2.5
2.5
3.5
2.5
4.0
1.5
2.0
2.5
2.5
2.0
3.5
2.5
1.2
2.4
laion/clap-htsat-fused
3.0
2.5
3.0
5.0
3.0
1.5
3.0
3.0
2.5
3.0
2.0
1.5
3.5
3.0
5.0
3.5
3.0
4.0
2.5
5.5
2.5
3.5
4.0
2.5
2.0
2.5
2.5
1.4
2.0
Appendix
Table 47: CommonVoiceMini21 A2T retrieval results (languages 30–58 of 114). Best per language in bold.
Model
luo
lv
mdf
mhr
mk
ml
mn
mr
mrj
mt
myv
nan
ne
nl
nn
oc
or
os
pa
pl
ps
pt
rm
ro
ru
rw
sah
sat
sc
LCO-Embedding/LCO-Embedding-Omni-3B
86.0
40.6
92.5
79.0
78.2
55.8
16.2
78.0
74.4
80.0
88.0
59.2
61.4
98.8
66.5
90.9
63.1
66.9
76.6
87.0
50.8
98.6
98.4
85.0
99.0
46.5
57.5
4.4
94.5
LCO-Embedding/LCO-Embedding-Omni-7B
75.0
35.8
86.0
65.4
74.0
39.8
10.4
79.0
57.0
77.2
81.3
61.8
48.5
99.4
53.6
89.4
58.8
59.2
78.6
66.6
47.6
98.8
97.9
76.6
99.0
36.5
43.0
6.2
93.5
Qwen/Qwen2-Audio-7B
4.5
2.0
7.5
1.8
1.0
2.6
2.0
1.4
3.0
2.2
3.2
1.2
3.7
1.6
1.7
4.7
3.3
7.7
1.6
2.2
2.0
3.8
3.5
2.4
4.2
4.0
5.0
8.8
5.0
microsoft/speecht5_multimodal
4.0
1.0
4.7
1.2
0.8
1.0
1.0
0.4
1.4
1.4
1.9
0.6
2.2
2.6
1.9
1.8
0.7
1.5
0.8
1.0
1.0
2.0
3.0
1.2
1.0
3.5
2.0
5.3
7.0
laion/clap-htsat-unfused
4.5
1.0
8.4
1.4
0.4
2.0
1.6
1.2
0.6
1.4
0.8
0.8
2.2
0.8
1.5
2.2
1.4
3.8
1.0
1.0
1.4
1.2
2.5
0.8
0.4
4.0
3.0
5.3
2.5
laion/clap-htsat-fused
3.5
1.6
5.6
2.2
2.0
1.0
1.8
0.8
1.8
0.8
2.4
1.6
2.9
1.0
1.9
2.2
2.1
5.4
1.6
1.0
1.2
1.6
0.5
1.4
1.0
3.0
4.0
3.5
4.0
Appendix
Table 48: CommonVoiceMini21 A2T retrieval results (languages 59–87 of 114). Best per language in bold.
Model
sk
skr
sl
sq
sr
sv
sw
ta
te
th
tig
tk
tn
tok
tr
tt
ug
uk
ur
uz
vi
yi
yo
yue
zgh
zh
zza
LCO-Embedding/LCO-Embedding-Omni-3B
87.5
61.0
70.0
79.0
62.0
78.0
52.0
56.5
90.2
93.5
59.0
55.5
49.5
85.5
88.0
54.0
66.5
93.0
91.5
66.5
86.5
61.5
49.0
98.5
62.0
99.5
69.0
LCO-Embedding/LCO-Embedding-Omni-7B
82.5
62.0
70.5
74.0
63.5
65.5
56.0
28.5
63.9
95.0
40.5
41.5
35.5
83.0
75.5
42.0
58.5
94.0
90.5
55.0
89.0
62.2
33.0
99.0
66.0
99.5
58.5
Qwen/Qwen2-Audio-7B
6.5
2.5
3.5
5.0
5.5
5.0
5.0
5.0
18.0
5.0
3.0
5.0
4.0
4.5
3.5
4.5
4.0
6.5
6.5
3.5
4.0
4.2
2.0
6.5
3.5
4.5
4.5
microsoft/speecht5_multimodal
4.0
2.5
4.5
3.0
3.0
4.0
4.5
4.0
8.2
3.0
3.5
3.0
4.5
3.0
5.0
3.0
1.5
2.5
4.5
2.0
1.0
2.1
1.0
2.5
2.5
2.5
3.0
laion/clap-htsat-unfused
3.0
4.0
2.5
4.5
4.5
3.5
2.5
2.5
8.2
2.0
3.0
2.5
1.0
3.0
3.5
2.5
2.0
4.5
1.5
3.5
2.0
4.9
4.0
2.5
2.0
2.5
2.5
laion/clap-htsat-fused
3.0
2.5
3.0
3.0
3.0
1.5
2.5
2.0
9.8
1.0
2.5
2.5
3.0
3.5
2.0
3.5
3.0
5.0
3.0
3.5
3.0
2.8
2.0
2.5
2.5
3.0
2.5
Appendix
Table 49: CommonVoiceMini21 A2T retrieval results (languages 88–114 of 114). Best per language in bold.
Model
ab
af
am
ar
as
ast
az
ba
bas
be
bg
bn
br
ca
ckb
cnh
cs
cv
cy
da
dav
de
dv
dyu
el
en
eo
es
et
LCO-Embedding/LCO-Embedding-Omni-3B
22.0
83.8
22.0
92.0
42.0
99.4
83.7
29.0
58.0
99.5
84.0
69.0
60.0
99.0
45.5
48.5
86.0
55.0
23.0
74.0
51.0
99.5
6.5
87.3
50.5
99.0
96.0
100.0
53.5
LCO-Embedding/LCO-Embedding-Omni-7B
19.5
83.8
22.5
91.0
28.0
96.2
79.3
29.5
48.0
98.0
75.5
62.0
54.0
98.0
32.5
42.5
72.5
53.5
28.5
60.5
33.0
99.5
3.5
81.0
23.5
98.5
91.5
100.0
52.0
Qwen/Qwen2-Audio-7B
5.5
10.3
6.5
4.0
2.5
7.6
5.4
3.5
4.0
5.5
5.5
2.5
5.0
7.5
6.0
4.5
3.0
2.0
2.5
2.5
4.0
6.5
2.0
7.9
4.0
11.0
4.5
8.0
2.5
microsoft/speecht5_multimodal
1.5
7.7
2.5
2.5
3.0
5.1
6.5
2.5
3.0
3.5
2.0
2.5
4.5
3.5
1.5
7.5
2.5
2.0
4.5
5.5
3.5
4.0
2.5
14.3
2.5
5.5
3.0
2.5
3.5
laion/clap-htsat-unfused
3.0
5.1
2.0
1.5
2.5
3.2
3.3
3.5
3.5
3.0
3.0
3.0
4.5
5.5
3.0
3.0
1.5
2.0
3.5
0.5
4.5
5.0
2.5
11.1
2.5
2.0
3.5
3.5
3.5
laion/clap-htsat-fused
2.5
6.0
1.5
1.5
4.0
2.5
2.2
2.0
5.0
1.5
2.0
2.5
3.5
3.5
2.5
3.0
3.5
1.0
1.5
2.5
4.5
1.0
2.0
15.9
3.5
2.0
5.0
1.5
3.5
Appendix
Table 50: CommonVoiceMini21 T2A retrieval results (languages 1–29 of 114). Best per language in bold.
Model
eu
fa
fi
fr
fy
ga
gl
gn
ha
he
hi
hsb
hu
hy
ia
id
it
ja
ka
kab
kk
kln
kmr
ko
ky
lg
lij
lt
ltg
LCO-Embedding/LCO-Embedding-Omni-3B
58.5
33.5
54.5
100.0
90.5
34.0
99.5
60.0
55.0
45.0
95.5
77.5
53.5
47.0
99.0
94.5
100.0
93.5
69.5
38.0
25.5
48.5
53.0
98.5
44.5
31.5
94.5
68.4
57.2
LCO-Embedding/LCO-Embedding-Omni-7B
58.5
30.5
33.5
100.0
89.5
20.5
100.0
47.5
44.5
29.5
94.5
72.0
48.5
27.5
98.5
96.0
100.0
92.0
62.0
35.5
20.0
36.5
43.5
97.5
43.0
24.5
90.5
58.4
47.0
Qwen/Qwen2-Audio-7B
4.0
2.0
4.5
5.0
4.0
3.5
8.5
4.0
5.0
2.0
5.0
4.0
4.5
3.5
7.0
4.5
8.0
2.0
4.0
3.5
3.5
3.5
4.5
5.0
4.0
7.0
2.5
1.6
1.8
microsoft/speecht5_multimodal
3.0
2.5
6.0
4.0
3.5
1.5
3.5
4.0
3.0
3.0
3.0
4.5
2.5
3.0
5.0
5.0
5.0
2.5
3.0
3.5
2.0
4.5
3.0
2.5
3.0
4.5
3.0
1.6
1.2
laion/clap-htsat-unfused
2.0
2.5
4.0
3.0
5.0
1.5
2.5
4.5
2.5
3.0
3.0
3.0
3.5
2.5
3.0
2.5
3.5
4.0
2.5
3.5
2.0
2.0
3.5
2.5
2.5
2.5
5.5
1.0
1.4
laion/clap-htsat-fused
4.0
2.5
3.5
3.5
4.0
4.0
3.0
5.5
2.0
2.0
1.5
5.5
2.0
2.5
2.5
4.0
3.0
4.0
2.5
4.5
2.5
2.5
2.5
4.0
3.0
1.5
5.0
1.6
1.0
Appendix
Table 51: CommonVoiceMini21 T2A retrieval results (languages 30–58 of 114). Best per language in bold.
Model
luo
lv
mdf
mhr
mk
ml
mn
mr
mrj
mt
myv
nan
ne
nl
nn
oc
or
os
pa
pl
ps
pt
rm
ro
ru
rw
sah
sat
sc
LCO-Embedding/LCO-Embedding-Omni-3B
61.5
47.0
76.6
50.4
78.2
49.0
8.0
77.6
43.4
74.6
68.3
60.8
50.7
98.4
67.5
91.2
56.0
45.4
67.6
89.4
38.2
97.8
96.8
79.2
98.6
26.5
25.5
3.5
94.5
LCO-Embedding/LCO-Embedding-Omni-7B
62.0
38.8
63.6
41.0
71.8
32.6
5.4
76.0
33.4
70.4
61.1
63.8
39.7
98.2
51.9
87.2
61.9
33.1
70.0
63.8
35.6
98.2
95.4
69.8
98.4
23.0
27.0
3.5
94.0
Qwen/Qwen2-Audio-7B
4.0
1.6
4.7
1.6
1.6
3.4
1.8
0.8
1.8
1.4
2.1
1.6
3.7
2.6
2.4
4.0
1.4
5.4
1.8
1.0
1.4
4.4
3.0
2.6
4.0
3.5
3.0
5.3
3.5
microsoft/speecht5_multimodal
6.5
1.6
4.7
0.8
1.2
1.0
0.4
1.0
1.0
1.8
1.1
1.6
1.8
2.4
1.9
1.1
1.0
2.3
1.0
1.2
1.2
1.2
1.8
1.8
1.0
4.0
1.5
4.4
4.0
laion/clap-htsat-unfused
3.5
1.2
8.4
0.6
1.4
1.2
0.6
1.4
1.4
1.2
1.9
1.6
1.8
1.2
2.2
2.6
1.2
4.6
0.8
2.4
1.4
2.2
1.2
1.6
1.0
3.5
2.0
7.1
5.0
laion/clap-htsat-fused
3.5
2.0
5.6
1.2
1.0
0.6
1.4
1.0
2.0
2.0
1.3
1.8
2.6
0.8
2.7
1.8
0.7
10.0
1.4
1.0
0.6
1.6
1.4
1.4
1.6
2.0
2.0
2.7
4.5
Appendix
Table 52: CommonVoiceMini21 T2A retrieval results (languages 59–87 of 114). Best per language in bold.
Model
sk
skr
sl
sq
sr
sv
sw
ta
te
th
tig
tk
tn
tok
tr
tt
ug
uk
ur
uz
vi
yi
yo
yue
zgh
zh
zza
LCO-Embedding/LCO-Embedding-Omni-3B
89.0
48.0
74.5
72.0
55.5
74.0
52.5
40.5
77.0
92.5
30.0
38.0
29.5
67.5
83.0
32.5
45.0
93.5
88.5
50.0
86.0
44.8
25.5
99.5
26.5
99.0
63.0
LCO-Embedding/LCO-Embedding-Omni-7B
84.5
45.0
75.0
71.0
52.5
63.0
51.5
24.0
57.4
94.5
24.0
32.5
20.5
62.5
76.5
33.0
41.5
92.5
90.5
43.5
87.5
52.4
10.5
99.5
30.0
99.0
50.5
Qwen/Qwen2-Audio-7B
2.5
3.5
4.5
5.0
6.5
4.5
5.0
2.5
14.8
3.0
3.5
3.5
5.0
4.5
3.5
2.0
5.0
6.0
4.5
4.0
5.0
4.9
1.5
6.0
3.5
6.5
2.5
microsoft/speecht5_multimodal
2.5
2.5
2.0
3.5
2.0
3.0
3.5
1.0
8.2
2.5
3.0
4.5
5.5
7.5
4.5
1.5
3.0
1.0
4.0
2.5
3.5
4.2
2.5
2.5
1.5
2.5
5.5
laion/clap-htsat-unfused
3.0
2.5
5.0
5.0
1.5
4.5
2.5
2.0
6.6
2.5
2.5
3.0
2.5
4.5
3.5
4.5
2.5
4.5
3.0
1.0
2.0
4.2
2.0
7.5
2.5
3.0
2.0
laion/clap-htsat-fused
3.0
3.0
4.0
3.0
4.5
2.0
2.5
1.0
8.2
2.5
1.5
2.5
3.5
5.0
4.5
1.5
3.0
2.5
1.5
4.0
2.5
5.6
2.0
2.5
1.5
2.5
5.5
Appendix
Table 53: CommonVoiceMini21 T2A retrieval results (languages 88–114 of 114). Best per language in bold.
Long-form audio performance is often summarized by context length and aggregate accuracy, obscuring how language, evidence, and task jointly shape difficulty. We introduce MuLA-Bench: 5,038 open-ended questions over 1,769 in-the-wild recordings totaling 1,377.9 hours, covering 16 languages and eight domains. A balanced Language x Domain semantic track supports controlled comparisons, while a complementary acoustic track preserves naturally occurring non-speech evidence. Evidence-grounded generation, shortcut checks, and language-expert review provide auditable questions without translating a shared source set or injecting target sounds. We evaluate ten audio-language models and conduct pooled diagnostics on a fixed eight-model cohort. Language rankings change across domains and tasks; acoustic-semantic performance gaps vary with the requested operation; and temporal errors can persist after the correct event is identified. Long-range retrieval is comparatively strong, while precise clock alignment and factual grounding of natural acoustic events remain fragile. MuLA-Bench thus exposes conditional failure patterns that a single long-context score does not capture.
Zeyu Yang, Xinyu Zhang, Zibo Bi +5
The Chinese University of Hong Kong, Shenzhen · Alibaba Token Hub, Alibaba Group · Xi’an Jiaotong University
We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: accuracy (semantic correctness), completeness (coverage of reference content), and hallucination (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini3Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research.
With the development of audio large language models (AudioLLMs), audio captioning needs to move from brief descriptions toward open-ended and fine-grained free-form descriptions. Existing evaluations often focus on generation quality or task performance, making it difficult to diagnose information coverage and description reliability. We propose MMAC, a Massive Multi-dimensional benchmark for Audio Captioning. MMAC contains 5,638 audio clips from more than 20 data sources, covering 6 capability categories and 15 evaluation dimensions. Given a model-generated caption, MMAC checks whether it mentions relevant information in the target dimension and whether the mentioned content is consistent with the reference label. We evaluate representative open-source and proprietary AudioLLMs. Results show clear differences across evaluation dimensions, information coverage, and description reliability. We will release the MMAC benchmark and evaluation code.
Weijie Wu, Junbo Li, Lin Li +2
School of Informatics, Xiamen University, China · DiDi Global Inc., Beijing, China · School of Electronic Science and Engineering, Xiamen University, China