AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

Jielin Qiu; Jianguo Zhang; Zixiang Chen; Liangwei Yang; Ming Zhu; Juntao Tan; Haolin Chen; Wenting Zhao

cs.SDFeb 27, 2026

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

Authors: Jielin Qiu, Jianguo Zhang, Zixiang Chen, Liangwei Yang, Ming Zhu, Juntao Tan, Haolin Chen, Wenting Zhao, +8 more

Organizations: Salesforce AI Research

Abstract

We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: accuracy (semantic correctness), completeness (coverage of reference content), and hallucination (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini3Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research.

Figures & tables

Figure 1: Distribution of audio duration (top) and reference caption length (bottom) for each category. Dashed lines indicate means. Music clips are longest; speech clips are shortest but have detailed emotional captions.

Figure 2: Overall LLM judge scores for all 13 models, colored by provider. Gemini models (blue) dominate the top positions, followed by OpenAI Chat Completions (green) and Realtime (orange).

Rank	Model	Accuracy	Complete.	Halluc.	Overall	METEOR	BLEU-4	ROUGE-L
1	gemini-3-pro-preview	6.18	6.39	5.42	6.00	0.234	0.037	0.152
2	gemini-2.5-pro	6.00	6.27	5.37	5.88	0.220	0.021	0.127
3	gemini-2.5-flash	5.78	5.78	5.66	5.74	0.225	0.031	0.141
4	gpt-4o-audio-preview	5.52	5.12	6.30	5.65	0.211	0.018	0.162
5	gemini-3-flash-preview	5.68	5.81	5.11	5.53	0.240	0.041	0.172
6	gemini-2.0-flash	5.01	4.43	6.47	5.30	0.174	0.025	0.188

Table 1: AudioCapBench results across 13 models on 1,000 samples. LLM judge scores are on a 0–10 scale. Bold = best; underline = second best. Sorted by Overall.

Model	Sound	Music	Speech
gemini-3-pro-preview	6.40	5.68	5.79
gemini-2.5-pro	5.76	5.66	6.26
gemini-2.5-flash	5.47	5.19	6.63
gpt-4o-audio-preview	5.40	5.15	6.47
gemini-3-flash-preview	5.72	4.88	5.92
gemini-2.0-flash	4.55	5.27	6.34

Table 2: LLM Overall scores by audio category (0–10). Bold = best per column.

Figure 3: Per-category LLM Overall scores. Speech is consistently the easiest category across all models, while music is the hardest. Performance gaps between models are largest on sound captioning.

Figure 4: Accuracy vs. hallucination trade-off. Models in the upper-right are both accurate and grounded. OpenAI mini models cluster in the upper-left (conservative but empty). Gemini models tend toward higher accuracy but lower hallucination scores.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Category	Source	Samples	Avg Caption
Sound	Clotho v2 test	200	90 chars
Sound	AudioCaps test	200	104 chars
Music	MusicCaps eval	300	264 chars
Speech	Emo Speech Caption	300	182 chars
Total		1,000

Appendix

Table 3: Evaluation dataset composition.

Explore similar work

MMAC: A Massive Multi-dimensional Benchmark for Audio Captioning
Jul 29, 2026Weijie Wu, Junbo Li, Lin Li +2Audio-Language Model Evaluation Audio Captioning
School of Informatics, Xiamen University, China · DiDi Global Inc., Beijing, China · School of Electronic Science and Engineering, Xiamen University, China
Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
Feb 5, 2026Jinchuan Tian, Haoran Wang, Bo-Hao Su +14Audio Understanding Audio-Language Models
1Language Technologies Institute, Carnegie Mellon University · 2LY Corporation · 3NVIDIA Research
TAC: Timestamped Audio Captioning
Feb 17, 2026Sonal Kumar, Prem Seetharaman, Ke Chen +8Audio Captioning Temporal Audio Grounding
University of Maryland, College Park, USA · Adobe Research, USA · OpenAI, USA (work done while at Adobe).

Jul 29, 2026J/K move · Enter open · S save

MMAC: A Massive Multi-dimensional Benchmark for Audio Captioning

Weijie Wu, Junbo Li, Lin Li, Jun Fang+1

School of Informatics, Xiamen University, China · DiDi Global Inc., Beijing, China · School of Electronic Science and Engineering, Xiamen University, China

Audio-Language Model Evaluation Audio Captioning

Open paper AskPDF arXiv

With the development of audio large language models (AudioLLMs), audio captioning needs to move from brief descriptions toward open-ended and fine-grained free-form descriptions. Existing evaluations often focus on generation quality or task performance, making it difficult to diagnose information coverage and description reliability. We propose MMAC, a Massive Multi-dimensional benchmark for Audio Captioning. MMAC contains 5,638 audio clips from more than 20 data sources, covering 6 capability categories and 15 evaluation dimensions. Given a model-generated caption, MMAC checks whether it mentions relevant information in the target dimension and whether the mentioned content is consistent with the reference label. We evaluate representative open-source and proprietary AudioLLMs. Results show clear differences across evaluation dimensions, information coverage, and description reliability. We will release the MMAC benchmark and evaluation code.

Abstract

Explore similar work

MMAC: A Massive Multi-dimensional Benchmark for Audio Captioning

Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions

TAC: Timestamped Audio Captioning