AT-Differ: Dataset for Audio-Text Difference Captioning

Akihiro Fujii; Yuki Okamoto; Yuki Saito; Hiroshi Saruwatari

cs.SDOct 21, 2026

AT-Differ: Dataset for Audio-Text Difference Captioning

Authors: Akihiro Fujii, Yuki Okamoto, Yuki Saito, Hiroshi Saruwatari

Organizations: The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan

Abstract

Evaluating audio-text semantic alignment is important in various audio-language tasks. Existing methods mainly quantify audio-text semantic alignment using a single score and cannot explicitly describe how audio and text differ. In this paper, we propose audio-text difference captioning, a task that generates textual descriptions of differences between audio and text. We also construct a new dataset AT-DIFFER containing audio-text pairs with systematically simulated differences and corresponding difference captions. Using AT-DIFFER, we fine-tune an existing multimodal large language model to describe differences between audio and text. The experimental results show that the original model has limited ability to describe such differences, whereas fine-tuning on AT-DIFFER substantially improves this ability.

Figures & tables

Figure 1: Overview of audio–text difference captioning.

Audio structure	Difference type	Difference caption
Single, Sim., & Seq. (common)	Semantic difference	The caption mistakes the sound of [ correct event ] for the sound of [ wrong event ].
	Text omission	The caption leaves out the sound of [ absent event ] that is actually present.
	Sound omission	The caption adds the sound of [ nonexistent event ] that is not actually there.
Seq. only	Seq. discrepancy	The caption reverses the order of [ event A ] and [ event B ].
Seq. only	Sim. discrepancy	The caption describes the sequential [ event A ] and [ event B ] as overlapping.
Sim. only	Sim. discrepancy	The caption describes the simultaneous [ event A ] and [ event B ] as sequential.

Table 1: Templates of audio–text difference caption. “Sim.” and “Seq.” denote “Simultaneous” and “Sequential,” respectively.

Figure 2: Procedure for constructing AT-DIFFER.

Figure 3: Model architecture and fine-tuning strategies. Parameters of self-attention projection matrices are fine-tuned using LoRA. Other parameters, such as those of encoders and Mixture-of-Experts feed-forward network (MoE FFN), are frozen.

Figure 4: The abridged prompt and an example of model output.

Model	BLEU $\uparrow$				METEOR $\uparrow$	ROUGE-L $\uparrow$	CIDEr-D $\uparrow$	SPICE $\uparrow$	SPIDEr $\uparrow$	BERTScore $\uparrow$			SBERT $\uparrow$
Model	1	2	3	4	METEOR $\uparrow$	ROUGE-L $\uparrow$	CIDEr-D $\uparrow$	SPICE $\uparrow$	SPIDEr $\uparrow$	P	R	F1	SBERT $\uparrow$
Baseline	0.550	0.429	0.337	0.269	0.273	0.498	1.021	0.449	0.735	0.545	0.539	0.542	0.735
Random
Fixed audio structure & diff. type	0.768	0.687	0.614	0.539	0.370	0.779	0.892	0.377	0.634	0.676	0.672	0.675	0.627
Fixed sound event	0.675	0.544	0.444	0.376	0.341	0.652	4.839	0.480	2.659	0.631	0.612	0.622	0.864
Method 1 (Difference caption only)	0.618	0.558	0.506	0.469	0.339	0.618	2.858	0.469	1.664	0.643	0.600	0.620	0.755

Table 2: Results of audio–text difference captioning. Bold indicates the best performance for each metric among the models.

Figure 5: Confusion matrices for difference type classification. The matrices cover only samples annotated with at most one difference type; predictions assigning two or more types are counted as “other.”

Audio-Text Alignment

See all 21 papers

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech
Feb 27, 2026Jielin Qiu, Jianguo Zhang, Zixiang Chen +13Audio-Language Model Evaluation Audio Captioning
Salesforce AI Research
Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons
Oct 20, 2026Tzu-Quan Lin, Chih-Kai Yang, Yi-Cheng Lin +2Audio-Text Alignment Audio-Language Models
Centre for Speech Technology Research, University of Edinburgh, United Kingdom · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan
Semantic Refinement of Universal Audio Representations through Audio-Description Alignment
Sep 8, 2026Lejun Min, Junyu Dai, Ruichen Zheng +7Audio-Text Alignment Audio Representation Learning
1Alibaba Token Foundry

Feb 27, 2026J/K move · Enter open · S save

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

Jielin Qiu, Jianguo Zhang, Zixiang Chen, Liangwei Yang+12

Salesforce AI Research

Audio-Language Model Evaluation Audio Captioning

Open paper AskPDF arXiv

We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: accuracy (semantic correctness), completeness (coverage of reference content), and hallucination (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini3Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research.

Abstract

Audio-Text Alignment

Explore similar work

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons

Semantic Refinement of Universal Audio Representations through Audio-Description Alignment