AVSD-Scenes: A Dataset for Audio-Visual Description of Urban Scenes

Dhanunjaya Varma Devalraju; Arshdeep Singh; Mark D. Plumbley

cs.AIOct 1, 2026

AVSD-Scenes: A Dataset for Audio-Visual Description of Urban Scenes

Authors: Dhanunjaya Varma Devalraju, Arshdeep Singh, Mark D. Plumbley

Organizations: IIT Madras, India · King’s College London, United Kingdom

Abstract

Natural language descriptions can provide rich semantic representations of audio-visual urban scenes, yet datasets that jointly describe both auditory and visual information remain limited. In this paper, we introduce AVSD-Scenes, a paired audio-visual scene description dataset for urban environments. The dataset contains 12,291 audio-visual scene descriptions generated from the TAU Urban Audio-Visual Scenes dataset. To construct the dataset, we first generate audio- and visual-based descriptions using Qwen2-Audio-7B and Qwen2.5-VL-7B, respectively. These modality-specific descriptions are then combined using large language models, namely Qwen3-14B, Mistral-Small-3.2-24B-Instruct-2506, and Gemma-3-27B-it, to produce multimodal descriptions that capture complementary information from both modalities. We benchmark AVSD-Scenes using semantic alignment, cross-modal retrieval, scene classification, LLM-as-a-judge evaluation, and human subjective assessment. Results show that multimodal descriptions improve semantic alignment and cross-modal retrieval performance compared with modality-specific descriptions while preserving strong scene-discriminative information. The generated descriptions achieve up to 94.5% accuracy in urban scene classification, while combining audio, visual, and description embeddings further improves accuracy to 95.4%. Furthermore, the descriptions remain highly scene-discriminative even when scene labels are removed from the prompting instructions, indicating that they capture semantic information derived from the audio-visual content rather than merely reflecting label information.

Figures & tables

Figure 1: Comparison of AVSD-Scenes descriptions with existing descriptions of an audio-visual scene example “park-stockholm-246-7354” taken from TAU Urban Audio-Visual Scenes 2021 development set [ 15 ] .

Figure 2: Dataset construction pipeline. All the models are loaded using 4-bit NormalFloat quantization with double quantization and FP16 computation for efficient inference.

Description	CLIP / CLAP			IMAGEBIND
Description	Text–Visual	Text–Audio	Avg.	Text–Visual	Text–Audio	Avg.
Modality-Specific	.3299	.2143	.2721	.3627	.0556	.2092
Multimodal (Qwen3)	.3187	.2854	.3021	.3425	.1696	.2561
Multimodal (Mistral)	.3018	.2613	.2816	.3165	.1615	.2390
Multimodal (Gemma)	.3083	.2735	.2909	.3218	.1599	.2409

Table 1: Cross-modal similarity of the generated descriptions. Avg. denotes the mean of the text–visual and text–audio similarity.

Description	CLAP / CLIP									IMAGEBIND
	T $\rightarrow$ A			T $\rightarrow$ V			Avg.			T $\rightarrow$ A			T $\rightarrow$ V			Avg.
	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Modality-Specific	24.99	49.97	60.85	85.19	94.57	97.13	55.09	72.27	78.99	24.40	40.95	51.82	90.76	97.01	98.39	57.58	68.98	75.11
Multimodal (Qwen3)	41.43	68.00	77.06	84.49	93.98	96.70	62.96	80.99	86.88	58.55	82.12	89.87	90.56	96.89	98.23	74.56	89.51	94.05
Multimodal (Mistral)	37.00	63.82	73.20	85.47	94.91	97.35	61.23	79.36	85.27	59.78	82.52	90.23	89.57	96.72	98.23	74.68	89.62	94.23
Multimodal (Gemma)	38.35	65.74	75.40	83.04	93.50	96.36	60.69	79.62	85.88	56.46	78.66	86.05	87.78	95.17	97.25	72.12	86.92	91.65

Table 2: Scene-level cross-modal retrieval performance (%). Avg. denotes the mean of text–audio (T

\rightarrow

A) and text–visual (T

\rightarrow

V) retrieval.

Figure 3: SVM-based urban scene classification accuracy for different combinations of audio (A), visual (V), and multimodal description (D) obtained using Qwen3, Mistral, and Gemma models.

Figure 4: t-SNE visualizations of audio, visual, and Mistral-generated scene-description embeddings, and their multimodal combinations.

Figure 5: LLM-as-a-Judge evaluation of multimodal descriptions generated by different models. Scores are reported on a five point scale (higher is better). AF: audio fidelity; VF: visual fidelity; EC: event coverage; Halluc.: hallucination; CMC: cross-modal consistency; Gram.: grammar; Flue.: fluency.

Description	CLAP T $\rightarrow$ A			IMAGEBIND T $\rightarrow$ A			IMAGEBIND T $\rightarrow$ V
Description	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Multimodal (Qwen3)	0.24	0.83	1.42	0.07	0.39	0.81	3.91	15.15	24.77
Multimodal (Mistral)	0.22	0.83	1.53	0.10	0.41	0.89	3.87	14.06	22.55
Multimodal (Gemma)	0.24	0.91	1.57	0.11	0.41	0.73	3.02	11.83	19.35

Table 3: Instance-level cross-modal retrieval performance (%).

Figure 6: SVM-based scene classification accuracy without scene-labels in the instructions. A: Audio, V: Visual, D: Description.

Explore similar work

Aug 10, 2026cs.CL

REFRAMED: Towards Realistic Audio Description Generation for Movies

Audio Description (AD) is a verbal narration of key visual content in videos, enabling access for visually impaired audiences. Unlike standard video captioning, AD is a structured editorial task: descriptions must be inserted into gaps in dialogue and must convey only what is needed to understand the narrative being told. However, existing approaches formulate AD generation in an artificial setting where both the content and timing of descriptions are pre-specified, reducing the task to clip-level captioning. They further rely on noisy transcription and alignment pipelines, and lack the rich parallel data required for modeling narrative context. We introduce a new formulation of AD generation in which models must jointly decide what to describe and when to do it. To support this, we present REFRAMED, a high-quality dataset of 2,023 videos that span 3,302 scenes from 206 movies, with professional AD transcripts (both American and British versions), professional subtitles and aligned screenplays. We also provide a manually curated challenge set that pairs full movies with multiple AD references, together with evaluation protocols that leverage dialogue gaps and multi-reference comparisons. Experiments with state-of-the-art AD systems and multimodal LLMs show that they outperform trivial baselines but fall far short of expert human performance. Our dataset and benchmark establish a new foundation for research on video understanding.

Igor Sterner, Mirella Lapata, Alex Lascarides +1

School of Informatics University of Edinburgh United Kingdom

Audio Understanding Films Multimodal Large Language Models

Ask PDF

Jul 11, 2026cs.LG

Empowering Long-form Omni-modal Understanding with Robust Audio Perception

Recent advances in large-scale multimodal models have drivenremarkable progress in vision-language tasks; however, comprehensiveomni-modal understanding remains under-explored, largely due to thescarcity of datasets with rich, explicitly aligned auditory cues. To bridgethis gap, we present AVDC (Audio-Visual Decoupled Captions), a large-scaledataset designed to disentangle visual and auditory semantics. Specifi-cally, we propose an automated pipeline that leverages off-the-shelf mod-els to annotate videos with tripartite captions: visual-only (V), audio-only (A), and joint audio-visual (AV). This decoupled structure explic-itly captures both modality-specific nuances and complex cross-modalinteractions. Building upon this, we introduce AVDC-QA-CoT, a Chain-of-Thought augmented question-answering dataset to foster audio-visualreasoning. To fully exploit these resources, we employ a two-stage train-ing paradigm: omni-modal caption generation pre-training on AVDC, fol-lowed by instruction tuning on AVDC-QA-CoT. Extensive experiments acrossdiverse downstream tasks, spanning video captioning, audio-centric anal-ysis, and omni-modal benchmarks, demonstrate consistent and signifi-cant performance gains, showing the efficacy of our proposed datasetsand training strategy in advancing omni-modal perception. Code anddataset are related on https://radiant0726.github.io/AVDC-web/.

Kaiying Yan, Luoyi Sun, Xiao Zhou +1

SAI, Shanghai Jiao Tong University, shanghai, China · Zhejiang University, Hangzhou, Zhejiang, china · Shanghai AI Lab, shanghai, China

Audio-Visual Reasoning Omni-Modal Multimodal Understanding Large Multimodal Models Video Captioning Audio Understanding

Ask PDF

May 27, 2026cs.SD

Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text

Audio generation has long been fragmented, with speech, music, and sound effects produced by domain-specific models that fail to jointly generate coherent audio scenes from a single description. The key obstacles are insufficient fine-grained supervision for real-world mixed audio and limited acoustic representations for modeling concurrent audio components. We present Dasheng AudioGen, a unified framework for generating general mixed-audio scenes from text. Dasheng AudioGen introduces structured multi-view captions, which explicitly decouple complex acoustic scenes into complementary description views, thereby enabling fine-grained control over audio layers. Furthermore, we employ a high-dimensional unified semantic-acoustic representation as the shared latent space. It injects semantic priors that facilitate cross-modal training convergence, while its high-dimensional feature space provides sufficient capacity to disentangle and fuse concurrent audio components effectively. With these designs, a simple flow-matching DiT achieves high-quality end-to-end audio scene generation. We also establish a comprehensive evaluation pipeline for audio scene generation. Experiments demonstrate that Dasheng AudioGen achieves performance approaching real-world recordings in mixed-audio categories, while remaining competitive with specialized models in single-type generation tasks. Demos are available at https://nieeim.github.io/Dasheng-AudioGen-Web/.

Jiahao Mei, Heinrich Dinkel, Yadong Niu +7

1X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · 2MiLM Plus, Xiaomi Inc., Beijing, China

Modern Generative Audio Models Acoustic Latent Space Audio Understanding Few-Step Generation Text Analysis and Detection

Ask PDF