EVLF: Early Vision-Language Fusion for Generative Dataset Distillation

Wenqi Cai; Yawen Zou; Guang Li; Chunzhi Gu; Chao Zhang

cs.CVMar 8, 2026

EVLF: Early Vision-Language Fusion for Generative Dataset Distillation

Authors: Wenqi Cai, Yawen Zou, Guang Li, Chunzhi Gu, Chao Zhang

Organizations: University of Toyama · Hokkaido University · University of Fukui

Abstract

Dataset distillation (DD) aims to synthesize compact training sets that enable models to achieve high accuracy with significantly fewer samples. Recent diffusion-based DD methods commonly introduce semantic guidance through late-stage cross-attention, where textual prompts tend to dominate the generative process. Although this strategy enforces label relevance, it diminishes the contribution of visual latents, resulting in over-corrected samples that mirror prompt patterns rather than reflecting intrinsic visual features. To solve this problem, we introduce an Early Vision-Language Fusion (EVLF) method that aligns textual and visual embeddings at the transition between the encoder and the generative backbone. By incorporating a lightweight cross-attention module at this transition, the early representations simultaneously encode local textures and global semantic directions across the denoising process. Importantly, EVLF is plug-and-play and can be easily integrated into any diffusion-based dataset distillation pipeline with an encoder. It works across different denoiser architectures and sampling schedules without any task-specific modifications. Extensive experiments demonstrate that EVLF generates semantically faithful and visually coherent synthetic data, yielding consistent improvements in downstream classification accuracy across varied settings. Source code is available at https://github.com/wenqi-cai297/earlyfusion-for-dd/.

Figures & tables

Figure 1: Comparison between traditional late-fusion approaches and the proposed EVLF . (a) Late-fusion methods inject textual prompts during the denoising process, causing semantic signals to dominate visual latent representations. (b) EVLF introduces vision-language alignment before diffusion, allowing semantic cues and visual features to co-evolve throughout generation. (c) Synthetic samples on ImageNette (256 $\times$ 256). (d) Synthetic samples on CIFAR-10 (32 $\times$ 32). Rows display real images, late-fusion results, and EVLF results. EVLF produces samples with stronger label fidelity and more coherent visual details.

Figure 2: Overview of EVLF. Visual latents from a VAE and text embeddings from a text encoder are fused via cross-attention at the encoder-backbone interface. The fused embeddings are trained with $\mathcal{L}_{\mathrm{MSE}}$ for visual preservation and $\mathcal{L}_{\mathrm{InfoNCE}}$ for semantic alignment. Fused embeddings are clustered and decoded to produce the distilled synthetic dataset.

IPC (Ratio)

Test Model

Random

Herding

DiT

IDC-1

Minimax

D 4 M

D 4 M+EVLF

MGD 3

MGD 3 +EVLF

Full

10 (0.8%)

ConvNet-6

24.3\!\pm\!{\scriptstyle 1.1}

26.7\!\pm\!{\scriptstyle 0.7}

34.2\!\pm\!{\scriptstyle 1.1}

26.9\!\pm\!{\scriptstyle 1.2}

33.3\!\pm\!{\scriptstyle 1.3}

33.3\!\pm\!{\scriptstyle 1.7}

29.4\!\pm\!{\scriptstyle 0.9}

\underline{34.3\!\pm\!{\scriptstyle 2.4}}

33.5\!\pm\!{\scriptstyle 1.9}

\mathbf{34.9\!\pm\!{\scriptstyle 1.0}}

86.4\!\pm\!{\scriptstyle 0.2}

ResNetAP-10

29.4\!\pm\!{\scriptstyle 0.8}

32.0\!\pm\!{\scriptstyle 0.3}

34.7\!\pm\!{\scriptstyle 0.5}

30.3\!\pm\!{\scriptstyle 1.2}

\underline{39.1\!\pm\!{\scriptstyle 0.5}}

36.2\!\pm\!{\scriptstyle 3.2}

33.2\!\pm\!{\scriptstyle 2.1}

37.3\!\pm\!{\scriptstyle 0.7}

36.6\!\pm\!{\scriptstyle 0.9}

\mathbf{39.3\!\pm\!{\scriptstyle 0.3}}

87.5\!\pm\!{\scriptstyle 0.5}

ResNet-18

27.7\!\pm\!{\scriptstyle 0.9}

30.2\!\pm\!{\scriptstyle 1.2}

34.7\!\pm\!{\scriptstyle 0.4}

33.4\!\pm\!{\scriptstyle 0.7}

\underline{37.3\!\pm\!{\scriptstyle 0.2}}

35.7\!\pm\!{\scriptstyle 1.6}

32.3\!\pm\!{\scriptstyle 1.2}

35.9\!\pm\!{\scriptstyle 2.1}

35.1\!\pm\!{\scriptstyle 1.8}

\mathbf{38.5\!\pm\!{\scriptstyle 0.3}}

89.3\!\pm\!{\scriptstyle 1.2}

20 (1.6%)

ConvNet-6

29.1\!\pm\!{\scriptstyle 0.7}

29.5\!\pm\!{\scriptstyle 0.3}

36.1\!\pm\!{\scriptstyle 0.8}

29.9\!\pm\!{\scriptstyle 1.0}

35.5\!\pm\!{\scriptstyle 0.8}

37.3\!\pm\!{\scriptstyle 0.1}

34.0\!\pm\!{\scriptstyle 2.3}

\underline{40.1\!\pm\!{\scriptstyle 2.6}}

36.2\!\pm\!{\scriptstyle 1.6}

\mathbf{40.2\!\pm\!{\scriptstyle 0.5}}

86.4\!\pm\!{\scriptstyle 0.2}

ResNetAP-10

32.7\!\pm\!{\scriptstyle 0.4}

34.9\!\pm\!{\scriptstyle 0.1}

41.1\!\pm\!{\scriptstyle 0.8}

35.2\!\pm\!{\scriptstyle 0.6}

43.4\!\pm\!{\scriptstyle 0.3}

43.3\!\pm\!{\scriptstyle 2.7}

40.1\!\pm\!{\scriptstyle 1.6}

42.8\!\pm\!{\scriptstyle 0.2}

\underline{44.5\!\pm\!{\scriptstyle 2.8}}

\mathbf{45.1\!\pm\!{\scriptstyle 0.9}}

87.5\!\pm\!{\scriptstyle 0.5}

ResNet-18

29.7\!\pm\!{\scriptstyle 0.5}

32.2\!\pm\!{\scriptstyle 0.6}

40.5\!\pm\!{\scriptstyle 0.5}

29.8\!\pm\!{\scriptstyle 1.7}

38.6\!\pm\!{\scriptstyle 0.2}

\underline{41.8\!\pm\!{\scriptstyle 1.9}}

38.4\!\pm\!{\scriptstyle 1.1}

40.7\!\pm\!{\scriptstyle 1.3}

40.3\!\pm\!{\scriptstyle 2.5}

\mathbf{42.1\!\pm\!{\scriptstyle 0.3}}

89.3\!\pm\!{\scriptstyle 1.2}

Table 1: Dataset distillation results on ImageWoof across different IPC settings and test models. The best results in each row are in

\mathbf{bold}

, and the second-best are underlined .

	IPC	Random	DiT	DM	Minimax	D 4 M	D 4 M+EVLF	MGD 3	MGD 3 +EVLF
Nette	10	$54.2\!\pm\!{\scriptstyle 1.6}$	$59.1\!\pm\!{\scriptstyle 0.7}$	$60.8\!\pm\!{\scriptstyle 0.6}$	$57.7\!\pm\!{\scriptstyle 1.2}$	$60.9\!\pm\!{\scriptstyle 1.7}$	$\underline{65.8\!\pm\!{\scriptstyle 1.2}}$	$64.3\!\pm\!{\scriptstyle 1.0}$	$\mathbf{66.0\!\pm\!{\scriptstyle 1.6}}$
	20	$63.5\!\pm\!{\scriptstyle 0.5}$	$64.8\!\pm\!{\scriptstyle 1.2}$	$66.5\!\pm\!{\scriptstyle 1.1}$	$64.7\!\pm\!{\scriptstyle 0.8}$	$66.3\!\pm\!{\scriptstyle 1.3}$	$\underline{71.7\!\pm\!{\scriptstyle 0.5}}$	$69.2\!\pm\!{\scriptstyle 1.9}$	$\mathbf{72.5\!\pm\!{\scriptstyle 0.8}}$
	50	$76.1\!\pm\!{\scriptstyle 1.1}$	$73.3\!\pm\!{\scriptstyle 0.9}$	$76.2\!\pm\!{\scriptstyle 0.4}$	$73.9\!\pm\!{\scriptstyle 0.3}$	$77.7\!\pm\!{\scriptstyle 1.1}$	$\mathbf{79.7\!\pm\!{\scriptstyle 0.5}}$	$79.2\!\pm\!{\scriptstyle 1.9}$	$\underline{79.5\!\pm\!{\scriptstyle 0.4}}$
IDC	10	$48.1\!\pm\!{\scriptstyle 0.8}$	$54.1\!\pm\!{\scriptstyle 0.4}$	$52.8\!\pm\!{\scriptstyle 0.5}$	$51.9\!\pm\!{\scriptstyle 1.4}$	$47.7\!\pm\!{\scriptstyle 0.5}$	$\mathbf{57.3\!\pm\!{\scriptstyle 1.5}}$	$55.0\!\pm\!{\scriptstyle 2.3}$	$\underline{56.3\!\pm\!{\scriptstyle 1.5}}$
	20	$52.5\!\pm\!{\scriptstyle 0.9}$	$58.9\!\pm\!{\scriptstyle 0.2}$	$58.5\!\pm\!{\scriptstyle 0.4}$	$59.1\!\pm\!{\scriptstyle 3.7}$	$56.3\!\pm\!{\scriptstyle 0.7}$	$\underline{62.0\!\pm\!{\scriptstyle 0.7}}$	$61.7\!\pm\!{\scriptstyle 1.0}$	$\mathbf{64.1\!\pm\!{\scriptstyle 0.3}}$
	50	$68.1\!\pm\!{\scriptstyle 0.7}$	$64.3\!\pm\!{\scriptstyle 0.6}$	$69.1\!\pm\!{\scriptstyle 0.8}$	$69.4\!\pm\!{\scriptstyle 1.4}$	$67.8\!\pm\!{\scriptstyle 1.0}$	$\underline{72.1\!\pm\!{\scriptstyle 0.3}}$	$71.0\!\pm\!{\scriptstyle 0.9}$	$\mathbf{72.7\!\pm\!{\scriptstyle 1.1}}$

Table 2: Comparison of SOTA methods under various IPC settings on ImageNette and ImageIDC. All results are on ResNetAP-10. Best in bold , second best underlined .

Dataset	IPC	SRe 2 L	RDED	D 4 M	D 4 M+EVLF
CIFAR-10	10	$29.3\!\pm\!0.5$	$37.1\!\pm\!0.3$	$37.6\!\pm\!1.8$	$\mathbf{45.7\!\pm\!0.5}$
CIFAR-10	50	$45.0\!\pm\!0.7$	$62.1\!\pm\!0.1$	$71.7\!\pm\!1.2$	$\mathbf{73.5\!\pm\!0.7}$
CIFAR-100	10	$27.0\!\pm\!0.4$	$42.6\!\pm\!0.2$	$53.2\!\pm\!0.7$	$\mathbf{56.2\!\pm\!0.4}$
CIFAR-100	50	$50.2\!\pm\!0.4$	$62.6\!\pm\!0.1$	$66.0\!\pm\!0.2$	$\mathbf{66.8\!\pm\!0.1}$

Table 3: Performance comparison on CIFAR-10 and CIFAR-100.

Dataset	IPC	SRe 2 L	RDED	D 4 M	D 4 M+EVLF
Tiny-ImageNet	10	$16.1\!\pm\!0.2$	$41.9\!\pm\!0.2$	$42.5\!\pm\!0.4$	$\mathbf{49.2\!\pm\!0.4}$
Tiny-ImageNet	50	$41.1\!\pm\!0.4$	$58.2\!\pm\!0.1$	$55.8\!\pm\!0.1$	$\mathbf{58.5\!\pm\!0.1}$

Table 4: Performance comparison on Tiny-ImageNet.

Dataset	IPC	Accuracy (%)
ImageNet-1K	10	SRe 2 L	RDED	DiT	Minimax
		$21.3\!\pm\!0.6$	$42.0\!\pm\!0.1$	$39.6\!\pm\!0.4$	$44.3\!\pm\!0.5$
		D 4 M	D 4 M+EVLF	MGD 3	MGD 3 +EVLF
		$47.7\!\pm\!0.6$	$48.3\!\pm\!0.3$	$50.8\!\pm\!0.6$	$\mathbf{51.3\!\pm\!0.3}$
	50	SRe 2 L	RDED	DiT	Minimax
		$46.8\!\pm\!0.2$	$56.5\!\pm\!0.1$	$52.9\!\pm\!0.6$	$58.6\!\pm\!0.3$

Table 5: Performance comparison on ImageNet-1K.

Method	CIFAR-10	CIFAR-100	Dogs	Flowers
w/o pre	$88.66\!\pm\!0.09$	$66.62\!\pm\!0.32$	$24.59\!\pm\!0.46$	$59.39\!\pm\!0.29$
Random	$88.46\!\pm\!0.09$	$65.97\!\pm\!0.08$	$23.08\!\pm\!0.40$	$56.81\!\pm\!0.40$
FRePo	$87.88\!\pm\!0.20$	$65.23\!\pm\!0.47$	$22.05\!\pm\!0.45$	$52.50\!\pm\!0.51$
KRR-ST	$89.33\!\pm\!0.19$	$68.04\!\pm\!0.22$	$35.51\!\pm\!0.45$	$70.45\!\pm\!0.34$
MGD 3 +EVLF	$\mathbf{90.23\!\pm\!0.15}$	$\mathbf{69.21\!\pm\!0.07}$	$\mathbf{36.18\!\pm\!0.37}$	$\mathbf{76.03\!\pm\!0.53}$

Table 6: Transfer learning results on target datasets using models pretrained on the distilled ImageNet-1K dataset.

Method	Ablation Components		IPC
Method	FT.	CA.	10	20	50
D 4 M	–	–	$47.7\!\pm\!0.5$	$56.3\!\pm\!0.7$	$67.8\!\pm\!1.0$
D 4 M+EVLF	$\checkmark$ – $\checkmark$	– $\checkmark$ $\checkmark$	$54.1\!\pm\!0.7$ $51.1\!\pm\!2.5$ $\mathbf{57.3\!\pm\!1.5}$	$61.1\!\pm\!0.1$ $57.5\!\pm\!0.3$ $\mathbf{62.0\!\pm\!0.7}$	$70.3\!\pm\!0.9$ $69.1\!\pm\!2.3$ $\mathbf{72.1\!\pm\!0.3}$

Table 7: Ablation on the contributions of denoiser fine-tuning (FT.) and the CrossAttention module (CA.) within the D 4 M pipeline. Results are reported on ImageIDC with ResNetAP-10 under varying IPC settings.

Figure 3: t-SNE visualization of synthetic and real samples on four ImageNet-1K classes. D 4 M [ 32 ] and MGD 3 [ 5 ] produce synthetic samples that occupy limited regions of the real-data manifold. With EVLF, the synthesized samples cover a broader and more varied region, indicating improved diversity and distributional alignment.

Figure 4: Parameter Analysis of $\lambda_{1}$ on ImageIDC.

Figure 5: Visualization of synthesized images generated by D 4 M and our EVLF under low- and high-resolution settings. (a) Bird class from CIFAR-10, and (b) Beagle class from ImageNet-1K. EVLF produces samples with clearer structure, richer textures, and improved consistency with class semantics across different image scales.

Diffusion-Based Dataset Distillation

See all 14 papers

Apr 1, 2026

Learnability-Guided Diffusion for Dataset Distillation

Training machine learning models on massive datasets is expensive and time-consuming. Dataset distillation addresses this by creating a small synthetic dataset that achieves the same performance as the full dataset. Recent methods use diffusion models to generate distilled data, either by promoting diversity or matching training gradients. However, existing approaches produce redundant training signals, where samples convey overlapping information. Empirically, disjoint subsets of distilled datasets capture 80-90% overlapping signals. This redundancy stems from optimizing visual diversity or average training dynamics without accounting for similarity across samples, leading to datasets where multiple samples share similar information rather than complementary knowledge. We propose learnability-driven dataset distillation, which constructs synthetic datasets incrementally through successive stages. Starting from a small set, we train a model and generate new samples guided by learnability scores that identify what the current model can learn from, creating an adaptive curriculum. We introduce Learnability-Guided Diffusion (LGD), which balances training utility for the current model with validity under a reference model to generate curriculum-aligned samples. Our approach reduces redundancy by 39.1%, promotes specialization across training stages, and achieves state-of-the-art results on ImageNet-1K (60.1%), ImageNette (87.2%), and ImageWoof (72.9%). Our code is available on our project page https://jachansantiago.github.io/learnability-guided-distillation/.

Jeffrey A. Chan-Santiago, Mubarak Shah

Institute of Artificial Intelligence, University of Central Florida

Dataset Distillation Diffusion-Based Dataset Distillation

Ask PDF

Jul 28, 2026

Dataset Distillation Based on Saliency-Driven Prototype Alignment

Dataset distillation aims to synthesize compact datasets that can approximate the performance of full-data training while significantly reducing computational and storage costs. However, diffusion-based distillation methods often struggle to preserve structural coherence and generalization, especially in visually complex domains. This issue often stems from latent prototypes that are weakly aligned with class-discriminative regions and contaminated by irrelevant background, thereby degrading generation quality and generalization. To address this limitation, we propose a saliency-driven distillation framework that constructs class-discriminative latent prototypes to enhance representativeness and generalization. The framework proceeds in two stages: (1) ensemble Grad-CAM++ saliency is used to construct prototypes emphasizing class-discriminative regions, and (2) hard-prototype refinement is then applied to construct challenging yet class-consistent prototypes, thereby enhancing discriminability and diversity. Importantly, the diffusion backbones (e.g., LDM and DiT) remain frozen; only lightweight classifiers used for saliency extraction are trained. Extensive experiments across multiple benchmarks demonstrate consistent performance improvements over strong baselines. Code will be released.

Yawen Zou, Wenqi Cai, Guang Li +3

University of Toyama · Hokkaido University · University of Fukui

Diffusion-Based Dataset Distillation Prototype Learning Dataset Distillation Diffusion Models

Ask PDF

Aug 4, 2026

Self-Supervised Representation-Guided Generative Dataset Distillation

Dataset distillation compresses a large training set into a compact synthetic set while retaining its downstream utility. Most existing methods target randomly initialized networks, whereas modern vision systems often adapt frozen pretrained encoders with lightweight modules. Distilled samples should therefore preserve the discriminative geometry of the pretrained representation space, which existing generative objectives do not explicitly consider. We propose self-supervised representation-guided generative dataset distillation (SRG), a framework that translates the SSL geometry into diffusion guidance. Specifically, SRG constructs class-wise prototypes from real-image SSL representations and performs guidance through three SSL-space objectives for prototype alignment, inter-class discrimination, and intra-class assignment. During diffusion sampling, it adopts a stage-wise guidance strategy: early denoising is anchored to the latent of the real image whose SSL representation is nearest to the assigned prototype, whereas later denoising is guided by the SSL-space objectives. This division preserves the visual realism provided by the generative prior while progressively steering samples toward representative and class-discriminative regions of the SSL representation space. SRG consistently outperforms the evaluated generative baselines across multiple datasets and IPC settings. A cross-encoder evaluation further indicates transfer across pretrained representation spaces. These results demonstrate the effectiveness of representation-guided generation for dataset distillation with pretrained SSL models.

Mingzhuo Li, Guang Li, Linfeng Ye +4

Hokkaido University, Japan · University of Toronto, Canada · University of Tokyo, Japan

Diffusion Model Guidance Dataset Distillation Diffusion-Based Dataset Distillation

Ask PDF

Abstract

Diffusion-Based Dataset Distillation

Explore similar work

Learnability-Guided Diffusion for Dataset Distillation

Dataset Distillation Based on Saliency-Driven Prototype Alignment

Self-Supervised Representation-Guided Generative Dataset Distillation