X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation

Saumyaranjan Mohanty; Aravind Reddy; Konda Reddy Mopuri

cs.CVOct 9, 2026

X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation

Authors: Saumyaranjan Mohanty, Aravind Reddy, Konda Reddy Mopuri

Organizations: Department of Artificial Intelligence, Indian Institute of Technology Hyderabad · Centre for Responsible AI, Wadhwani School of Data Science & AI, Indian Institute of Technology Madras

Abstract

Audio-visual dataset distillation aims to synthesize a small dataset from a large paired audio-visual dataset, while nearly preserving the training utility for neural networks. Existing approaches such as DM and AVDD face two critical limitations: 1) they are memory-intensive, making them difficult to scale to complex architectures and higher samples-per-class (SPC) settings, and 2) they often fail to preserve strong cross-modal semantic correspondence. To tackle these issues, we propose X-AVDD, a cross-attention-based distillation method that explicitly captures inter-modal relationships during synthesis. By decoupling cross-modal alignment from joint optimization, X-AVDD is substantially more memory-efficient, reducing end-to-end synthesis cost by $\sim 8\times$ GPU-hours, while producing highly aligned synthetic data. Empirically, X-AVDD significantly outperforms other audio-visual distillation methods. On VGGSound, an audio-visual dataset of short YouTube clips, X-AVDD improves top-1 test accuracy from $8.2\%\rightarrow24.0\%$ at SPC= $10$ and $9.8\%\rightarrow26.5\%$ at SPC= $20$ . Furthermore, X-AVDD improves cross-architecture generalization (e.g., $25.5\%\rightarrow35.3\%$ for a ViT target on AVE dataset at SPC $=50$ ) and successfully scales to complex architectures such as AudioCLIP, achieving $62.3\%$ accuracy using only $5.65\%$ of the full dataset, against the full dataset accuracy of $72.4\%$ while completing in $9\times$ lower training time compared to the full dataset.

Figures & tables

Figure 1: Accuracy on the VGGSound dataset under different samples-per-class settings. X-AVDD consistently outperforms DM and AVDD ( Kushwaha et al., 2024 ) .

Figure 2: Modality gap (class-wise $\ell_{2}$ distance between audio and visual class centers in the feature space) on the AVE dataset, and datasets distilled by AVDD and X-AVDD using AudioCLIP network. X-AVDD achieves a consistently lower modality gap.

Figure 3: Overview of the bi-directional cross-attention mechanism for Audio-Visual event recognition.

			ConvNet			ResNet-18		ViT
Dataset	SPC	Ratio (%)	DM	AVDD	X-AVDD	SRe 2 L	X-AVDD	SRe 2 L	X-AVDD
VGGS-10K	1	0.11	36.5 $\pm$ 2.5	40.4 $\pm$ 1.8	37.6 $\pm$ 0.8	21.7 $\pm$ 0.6	30.0 $\pm$ 1.5	16.7 $\pm$ 0.9	20.2 $\pm$ 0.5
	10	1.13	43.8 $\pm$ 1.7	55.0 $\pm$ 1.7	57.2 $\pm$ 0.9	46.9 $\pm$ 2.0	54.5 $\pm$ 0.7	25.4 $\pm$ 0.5	30.0 $\pm$ 0.7
	20	2.26	49.0 $\pm$ 2.4	58.0 $\pm$ 1.7	62.7 $\pm$ 0.5	52.7 $\pm$ 0.5	61.5 $\pm$ 0.8	26.1 $\pm$ 1.8	35.9 $\pm$ 0.8
	50	5.65	-	-	66.1 $\pm$ 0.3	60.4 $\pm$ 0.9	66.7 $\pm$ 0.9	30.2 $\pm$ 1.2	42.1 $\pm$ 1.2
	Full		74.7			77.4		62.1

Table 1: Comparison of accuracy (%) on synthetic datasets for various architectures with the ensemble fusion method. For the ConvNet architecture, comparisons are carried out against DM and AVDD, while for ResNet-18 and ViT, comparisons are carried out against a multimodal implementation of SRe 2 L. SPC denotes samples per class. Bold indicates best performance within each architecture group. - indicates Out-of-Memory (OOM) errors.

SPC	Ratio (%)	Accuracy (%)	Train time (Hrs.)	Ratio (%)	Accuracy (%)	Train time (Hrs.)
	VGGSound-10K			AVE
1	0.11	24.8 $\pm$ 0.6	0.28	0.1	44.0 $\pm$ 1.2	0.43
10	1.13	33.7 $\pm$ 0.3	0.56	1.0	74.5 $\pm$ 0.9	1.31
20	2.26	46.8 $\pm$ 0.9	1.16	2.0	76.8 $\pm$ 1.1	2.66
50	5.65	62.3 $\pm$ 0.8	2.74	5.0	81.3 $\pm$ 0.9	6.43
Full	100	72.4 $\pm$ 0.7	25.30	100	85.6 $\pm$ 1.1	72.12

Table 2: Test accuracy obtained by synthetic data generated by X-AVDD on VGGSound-10K and AVE datasets with AudioCLIP architecture. Ratio is the percentage of the condensed image as compared to the full dataset. With a fraction of the time taken to train the model on the full dataset, distilled images can achieve comparable generalization accuracy, underscoring the role of distillation for training audio-visual datasets with SOTA architectures.

Target	SPC	SRe 2 L	X-AVDD	SRe 2 L	X-AVDD	SRe 2 L	X-AVDD
		VGGSound-10K		AVE		Music-21
VGG-11	1	14.6 $\pm$ 3.7	20.4 $\pm$ 2.3	6.2 $\pm$ 2.5	11.4 $\pm$ 0.7	18.2 $\pm$ 1.9	22.3 $\pm$ 1.2
	10	34.8 $\pm$ 1.3	41.3 $\pm$ 2.2	21.8 $\pm$ 0.4	29.9 $\pm$ 0.7	39.7 $\pm$ 1.6	47.3 $\pm$ 2.6
	20	44.0 $\pm$ 2.4	51.5 $\pm$ 0.8	29.1 $\pm$ 0.3	35.6 $\pm$ 0.6	42.8 $\pm$ 1.1	58.7 $\pm$ 1.9
	50	53.2 $\pm$ 0.2	59.4 $\pm$ 0.2	35.0 $\pm$ 0.7	42.4 $\pm$ 0.3	48.5 $\pm$ 0.5	64.3 $\pm$ 1.1
ViT	1	13.9 $\pm$ 1.1	23.3 $\pm$ 0.9	12.6 $\pm$ 0.8	13.9 $\pm$ 0.1	10.0 $\pm$ 0.5	14.4 $\pm$ 1.8

Table 3: Comparative analysis of cross-architecture generalization performance of the distilled dataset generated using the ResNet-18 architecture for both audio and video modality. We experimented with two target architectures: VGG11 and ViT. For both, the synthetic dataset generated by X-AVDD achieves higher generalization accuracy than that generated by the SRe 2 L adaptation. Results are provided for the Ensemble fusion method.

SPC	AVDD	X-AVDD (C)	SRe 2 L	X-AVDD (R)	AVDD	X-AVDD (C)	SRe 2 L	X-AVDD (R)
	ImageBind				AST + SigLIP2
1	20.1 $\pm$ 0.4	25.7 $\pm$ 0.8	25.2 $\pm$ 0.6	28.3 $\pm$ 0.4	23.1 $\pm$ 1.3	24.3 $\pm$ 0.3	18.1 $\pm$ 1.2	37.3 $\pm$ 0.5
10	26.4 $\pm$ 0.9	40.9 $\pm$ 0.6	40.1 $\pm$ 0.8	58.4 $\pm$ 0.5	36.1 $\pm$ 1.0	42.2 $\pm$ 0.7	49.8 $\pm$ 0.7	58.7 $\pm$ 0.9
20	33.2 $\pm$ 0.5	43.1 $\pm$ 0.4	43.7 $\pm$ 0.3	64.3 $\pm$ 0.5	37.7 $\pm$ 1.3	49.1 $\pm$ 1.6	51.2 $\pm$ 1.2	65.6 $\pm$ 0.6
50	-	48.3 $\pm$ 0.3	56.7 $\pm$ 0.6	67.1 $\pm$ 0.3	-	51.7 $\pm$ 1.1	56.5 $\pm$ 0.7	73.5 $\pm$ 0.3
Full	79.8				81.3

Table 4: Performance analysis of distilled dataset for transfer learning with frozen backbone of SOTA models. X-AVDD (C) and X-AVDD (R) denote the synthetic datasets generated by ConvNet and ResNet-

18

, respectively. With only

50

samples per class, synthetic data generated by the ResNet-18 architecture and X-AVDD achieves performance comparable to that of training on the full dataset.

Method	VGGSound-10K	AVE	MUSIC-21	VGGSound
AVDD	165	471	363	5342
X-AVDD	23	51	45	628

Table 5: GPU-Hour comparison between AVDD and our proposed method for various datasets.

Panel of Figure 4 — Figure 4: Impact of $\alpha$ on test accuracy. The left panel shows results for the VGGSound- $10$ K dataset and the ResNet-18 architecture, while the right panel shows results for AVE and ResNet-18.

Table 10

Table 11

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$\mathcal{V}$	Original large-scale training dataset
$\mathcal{S}$	Distilled synthetic dataset
$\tau_{a}$	Tokenization operator for audio data
$\tau_{v}$	Tokenization operator for image data
$\mathcal{T}_{a}$	Audio $\rightarrow$ image branch of bi-directional cross-attention block
$\mathcal{T}_{v}$	Image $\rightarrow$ audio branch of bi-directional cross-attention block

Appendix

Table 10: Notation

Dataset	Method	Vendi $\uparrow$	Similarity $\downarrow$	Coverage $\uparrow$
VGGSound-10k	SRe 2 L	1.26	0.89	0.05
	X-AVDD	1.65	0.87	0.15
Music-21	SRe 2 L	2.13	0.81	0.03
	X-AVDD	3.04	0.77	0.13
VGGSound	SRe 2 L	1.23	0.81	0.05
	X-AVDD	1.71	0.77	0.11

Appendix

Table 11: Comparative analysis of diversity metrics. Synthetic data generated by X-AVDD performs better across all three diversity metrics as compared to the baseline multimodal adaptation of SRe 2 L.

Description	Value
GPU	32 GB Tesla V100
OS	Ubuntu 18.04.1 LTS
Python	3.11.7
Pytorch	2.4.1
NumPy	1.24.3

Appendix

Table 12: Details of the computing infrastructure.

Table 15

AVDD		X-AVDD
Steps	Time	Steps	Time
Buffer creation	1 hr	teacher training & embedding storage	65 mins.
synthesis	9 hrs.	synthesis	12 mins.
Total	10 hrs.	Total	1.3 hrs.

Appendix

Table 15: Timing comparison of AVDD and our X-AVDD on VGGSound-10k dataset for synthesis of SPC=

20

. Kindly note that, while the synthesis stage in AVDD is a recurring process, in our proposed method, training the teacher network and storage of the embeddings are one-time requirements, and only the synthesis stage is the recurring step for generating synthetic samples across various SPC settings.

Figure 5: Comparison of modality gap observed in the real dataset, synthetic data generated by AVDD, and our proposed method. As shown, our proposed method achieves a lower modality gap consistently across all classes in the VGGSound-10K dataset.

Figure 6: Impact of temperature on test accuracy. Results are provided for VGGSound-10K with ResNet-18 for SPC= $10$ and SPC= $20$ . We can see that accuracy increases with temperature till it reaches a maximum and then decreases, signifying the importance of the temperature parameter during soft label generation.

Dataset	SPC= $1$	SRe 2 L	X-AVDD
VGGSound-10K	1	20.8 $\pm$ 1.5	26.8 $\pm$ 0.7
	10	47.2 $\pm$ 0.7	55.4 $\pm$ 0.7
	20	49.6 $\pm$ 0.2	65.1 $\pm$ 0.2
	50	64.4 $\pm$ 1.2	69.5 $\pm$ 0.3
AVE	1	18.0 $\pm$ 0.1	21.2 $\pm$ 0.4
	10	41.5 $\pm$ 0.6	51.6 $\pm$ 0.4

Appendix

Table 16: Test set accuracy comparison for various datasets with ‘concat’ fusion method.

Dataset Distillation

Dataset distillation compresses training data into a small synthetic or selected set that retains the information needed to train useful models.

See all 67 papers

Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention
Aug 16, 2026Shengchuan Gao, Teng Hu, Bohao Feng +4Audio-Video Generation Diffusion Model Inference Acceleration
Shanghai Jiao Tong University · Alibaba Token Hub, Alibaba Group · Alibaba Cloud Computing
CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation
Oct 22, 2026Lifan Jiang, Yibo Zhao, Senlin cheng +6Audio-Video Generation Diffusion Model Caching
State Key Lab of CAD&CG, Zhejiang University · Ant Group
Multimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesis
Feb 23, 2026Junhyeok Choi, Sangwoo Mo, Minwoo ChaeDataset Distillation Multimodal Dataset
Department of Industrial and Management Engineering, Pohang University of Science and Technology

Aug 16, 2026J/K move · Enter open · S save

Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention

Shengchuan Gao, Teng Hu, Bohao Feng, Luchen Li+3

Shanghai Jiao Tong University · Alibaba Token Hub, Alibaba Group · Alibaba Cloud Computing

Audio-Video Generation Diffusion Model Inference Acceleration Cross-Modal Attention Audio-Visual Synchronization

Open paper AskPDF arXiv

Recent audio-visual generation models can synthesize synchronized video and sound in a unified diffusion process, but their inference cost remains high because long video token sequences require repeated attention computation across denoising steps. A variety of acceleration techniques have been developed for video generation models, including low-bit quantization, attention sparsification, and feature caching. However, since these methods are originally designed for video generation, directly applying them to audio-visual models overlooks the interactions between the audio and video branches and may therefore disrupt audio-video synchronization. We present a synchronization-aware acceleration framework for efficient audio-visual generation. Our key observation is that bidirectional audio-video cross-attention reveals structured interactions between the two branches, with high responses often concentrated on a few sound-related visual and temporal regions. Guided by this interaction pattern, we introduce a protected sparse attention strategy that preserves high-fidelity computation for synchronization-critical tokens while sparsifying redundant attention interactions. By explicitly accounting for cross-modal dependence during acceleration, our method improves inference efficiency while keeping video quality, audio quality, and audio-video synchronization.

Abstract

Dataset Distillation

Explore similar work

Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention

CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation

Multimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesis