HIERAMP: Coarse-to-Fine Autoregressive Amplification for Generative Dataset Distillation

Lin Zhao; Xinru Jiang; Xi Xiao; Qihui Fan; Lei Lu; Yanzhi Wang; Xue Lin; Octavia Camps

cs.CVMar 6, 2026

HIERAMP: Coarse-to-Fine Autoregressive Amplification for Generative Dataset Distillation

Authors: Lin Zhao, Xinru Jiang, Xi Xiao, Qihui Fan, Lei Lu, Yanzhi Wang, Xue Lin, Octavia Camps,

Organizations: Northeastern University · University of Alabama at Birmingham · The Ohio State University

Abstract

Dataset distillation often prioritizes global semantic proximity when creating small surrogate datasets for original large-scale ones. However, object semantics are inherently hierarchical. For example, the position and appearance of a bird's eyes are constrained by the outline of its head. Global proximity alone fails to capture how object-relevant structures at different levels support recognition. In this work, we investigate the contributions of hierarchical semantics to effective distilled data. We leverage the vision autoregressive (VAR) model whose coarse-to-fine generation mirrors this hierarchy and propose HIERAMP to amplify semantics at different levels. At each VAR scale, we inject class tokens that dynamically identify salient regions and use their induced maps to guide amplification at that scale. This adds only marginal inference cost while steering synthesis toward discriminative parts and structures. Empirically, we find that semantic amplification leads to more diverse token choices in constructing coarse-scale object layouts. Conversely, at fine scales, the amplification concentrates token usage, increasing focus on object-related details. Across popular dataset distillation benchmarks, HIERAMP consistently improves validation performance without explicitly optimizing global proximity, demonstrating the importance of semantic amplification for effective dataset distillation.

Figures & tables

Figure 2: Overview of the HierAmp framework. Left: Scale-Restricted Class Token Attention Mask. The class token attends only to image tokens from the corresponding scale, with grey regions indicating blocked attention, producing a scale-specific semantic summary. Right: Multi-Scale Semantic Feature Amplification. The Amplify Algorithm selects the top attention positions from the class-token map at each scale and amplifies them via a positive logit bias, guiding the model to focus on semantically important features during decoding.

Dataset	IPC	ResNet-18					ResNet-101
Dataset	IPC	Minimax	D 3 HR	RDED	CaO 2	Ours	Minimax	D 3 HR	RDED	CaO 2	Ours
CIFAR-10	10	$-$	$41.3\pm 0.1$	$37.1\pm 0.3$	$39.0\pm 1.5$	$\mathbf{44.3}\pm\mathbf{0.6}$	$-$	$35.8\pm 0.6$	$33.7\pm 0.3$	$-$	$\mathbf{64.3}\pm\mathbf{0.9}$
CIFAR-10	50	$-$	$70.8\pm 0.5$	$62.1\pm 0.1$	$-$	$\mathbf{72.0}\pm\mathbf{0.1}$	$-$	$63.9\pm 0.4$	$51.6\pm 0.4$	$-$	$\mathbf{64.3}\pm\mathbf{0.9}$
CIFAR-100	10	$-$	$49.4\pm 0.2$	$42.6\pm 0.2$	$-$	$\mathbf{52.0}\pm\mathbf{0.1}$	$-$	$46.0\pm 0.5$	$41.1\pm 0.2$	$-$	$\mathbf{49.7}\pm\mathbf{0.2}$
CIFAR-100	50	$-$	$65.7\pm 0.3$	$62.6\pm 0.1$	$-$	$\mathbf{66.5}\pm\mathbf{0.2}$	$-$	$\mathbf{66.6}\pm\mathbf{0.2}$	$63.4\pm 0.3$	$-$	$66.1\pm 0.2$
ImageNet-Woof	1	$19.9\pm 0.2$	$-$	$20.8\pm 1.2$	$\mathbf{21.1}\pm\mathbf{0.6}$	$20.9\pm 0.8$	$17.7\pm 0.9$	$-$	$19.6\pm 1.8$	$21.2\pm 1.7$	$20.3\pm 1.4$

Table 1: Top-1 accuracy comparison with four SOTA methods . As in RDED [ 40 ] , all methods adopt ResNet-18 as the teacher and are trained on both ResNet-18 and ResNet-101. ‘–’ denotes missing results in the original paper.

Student\Teacher		ResNet-18	MobileNet-V2	EfficientNet-B0
ResNet-18	RDED	$42.3\pm 0.6$	$40.4\pm 0.1$	$36.6\pm 0.1$
	D 3 HR	${44.3}\pm{0.3}$	${42.3}\pm{0.7}$	${38.3}\pm{0.2}$
	Ours	$\mathbf{44.6}\pm\mathbf{0.5}$	$\mathbf{42.9}\pm\mathbf{0.3}$	$\mathbf{38.7}\pm\mathbf{0.1}$
MobileNet-V2	RDED	$34.4\pm 0.2$	$33.8\pm 0.6$	$28.7\pm 0.2$
	D 3 HR	${43.4}\pm{0.3}$	$\mathbf{46.4}\pm\mathbf{0.2}$	${37.8}\pm{0.4}$
	Ours	$\mathbf{46.2}\pm\mathbf{0.1}$	${45.8}\pm{0.3}$	$\mathbf{38.0}\pm\mathbf{0.2}$

Table 2: Cross-architecture performance on ImageNet-1K, IPC

=

10.

Figure 3: Impact of attention amplification strategy on token entropy and coverage on ImageNet-1K, IPC $=$ 50. The histogram shows the percentage of classes whose codebook token entropy and coverage increased, decreased, or remained unchanged after amplifying different stages. Amplifying attention at coarse and mid scales promotes diversity, while fine-scale amplification can concentrate attention.

Amp. - $\rho_{n}\%$	Coarse	Mid	Fine	Full
0 - 0%	$45.6\pm 0.3$
5 - 30%	$46.7\pm 0.2$	$46.3\pm 0.1$	$46.6\pm 0.3$	$46.9\pm 0.2$
5 - 70%	$47.0\pm 0.3$	$46.4\pm 0.2$	$46.2\pm 0.3$	$46.6\pm 0.2$
3 - 50%	$46.8\pm 0.2$	$46.7\pm 0.3$	$46.5\pm 0.1$	$47.1\pm 0.1$
7 - 50%	$47.3\pm 0.2$	$47.4\pm 0.3$	$46.3\pm 0.2$	$47.2\pm 0.3$
5 - 50%	$47.6\pm 0.3$	$46.9\pm 0.1$	$46.5\pm 0.2$	$47.6\pm 0.1$

Table 3: Effect of amplification combination across different stages on ImageNet-1K, IPC

=

10. Amp. indicates the amplification number, while

\rho_{n}\%

refers to the top

\rho_{n}\%

of the

m_{N}

regions selected for amplification.

C-M-F	0.1-5-5	5-0.1-5	5-5-0.1
Top-1 Acc. (%)	$45.9\pm 0.1$	$46.6\pm 0.1$	$46.6\pm 0.3$
C-M-F	0.5-5-5	5-0.5-5	5-5-0.5
Top-1 Acc. (%)	$46.3\pm 0.2$	$46.5\pm 0.3$	$46.8\pm 0.3$

Table 4: Ablation on amplification strength across stages on ImageNet-1K, IPC

=

10. Each row shows a different setting of Coarse-Mid-Fine (C-M-F) amplification numbers and accuracy.

Figure 4: Heatmap of unique codebook token occurrences across patch positions and scales on ImageNet-1K, Class 51 – Triceratops, IPC $=$ 50 . Darker patches indicate a higher number of unique tokens. The average of unique-token count for each scale is displayed in the upper-right corner of each heatmap.

Figure 5: Example generated images and attention heatmaps of the class token. Our method produces richer object details and quantities, achieves stronger semantic alignment, and enhances object-background dependence.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Model	IPC	DiT	DiT + HierAmp
ImageNet-Woof	ResNet-18	10	$41.0\pm 0.6$	$\mathbf{43.1}\pm\mathbf{0.3}$
ImageNet-Woof	ResNet-18	50	$66.2\pm 0.3$	$\mathbf{68.2}\pm\mathbf{0.3}$

Appendix

Table 5: Quantitative comparison of the DiT backbone before and after applying HIERAMP.

Figure 6: Qualitative comparison of the DiT backbone before and after applying HIERAMP.

IPC	Minimax	D 3 HR	VAR	Ours
10	$18.3\pm 0.2$	$19.0\pm 0.2$	$17.5\pm 0.1$	$\mathbf{17.3\pm 0.1}$
50	$14.3\pm 0.2$	$14.9\pm 0.1$	$\mathbf{13.1\pm 0.1}$	$13.2\pm 0.1$

Appendix

Table 6: FID of different dataset distillation methods on ImageNet-1K under 10 and 50 IPC.

IPC	VAR without class tokens	VAR with class tokens
10	$45.9\pm 0.3$	$45.6\pm 0.3$
50	$59.5\pm 0.1$	$59.3\pm 0.1$

Appendix

Table 7: Effect of class tokens on top-1 accuracy on ImageNet-1K. Models with and without class tokens show comparable performance across different IPC settings.

Figure 7: Visualization of images generated by VAR with and without class tokens on ImageNet-1K, IPC $=$ 10. “w/o” denotes VAR without class tokens, and “w/” denotes VAR with class tokens. The results indicate comparable generative capacity between the two models.

Model	Ours	D 3 HR (DDIM-based, 30 steps)
Time (s/img)	$\mathbf{0.147}\pm\mathbf{0.001}$	$0.456\pm 0.002$

Appendix

Table 8: Inference latency comparison with DDIM-based method on ImageNet-Woof.

Model	Base VAR	VAR + cls	VAR + cls + Amp (Ours)
Latency (s/img)	$0.139\pm 0.002$	$0.145\pm 0.002$	$0.147\pm 0.001$
Peak Memory (GB)	$1.770\pm 0.000$	$1.790\pm 0.000$	$1.840\pm 0.000$

Appendix

Table 9: Latency and peak memory for VAR-based distillation with incremental modules.

Figure 8: More amplification combinations impact of attention amplification strategy on token entropy on ImageNet-1K, IPC $=$ 50.

Figure 9: More amplification combinations impact of attention amplification strategy on token coverage on ImageNet-1K, IPC $=$ 50.

Figure 10: Visualization of the generated distilled images ( $224\times 224$ ) on ImageNet-1K, IPC $=$ 10. The first row shows distilled images without amplification (VAR with class tokens). The second row shows distilled images with amplification applied to Full stages (1-9) by a amplification factor of 5.

Figure 11: Visualization of the generated distilled images ( $224\times 224$ ) on ImageNet-1K, IPC $=$ 10. The first row shows distilled images without amplification (VAR with class tokens). The second row shows distilled images with amplification applied to Full stages (1-9) by a amplification factor of 5.

Dataset Distillation

Dataset distillation compresses training data into a small synthetic or selected set that retains the information needed to train useful models.

See all 67 papers

DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery
May 12, 2026Qianxin Xia, Zhiyong Shu, Wenbo Jiang +3Diffusion-Based Dataset Distillation Dataset Distillation
University of Electronic Science and Technology of China, Chengdu, China · Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore · Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore.
SAS: Semantic-aware Sampling for Generative Dataset Distillation
May 18, 2026Mingzhuo Li, Guang Li, Linfeng Ye +4Diversity-Aware Sampling Training Data Selection
Hokkaido University, N-14, W-9, Kita-Ku, Sapporo, 060-0814, Japan · University of Toronto, 27 King’s College Circle, Toronto, Ontario M5S 1A1, Canada · The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8654, Japan
Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment
Mar 30, 2026Tongfei Liu, Yufan Liu, Bing Li +1Dataset Distillation Dataset Condensation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chines Academy of Sciences, 95 Zhongguancun East Road, Beijing, 100190, Beijing, China. · School of Artificial Intelligence, University of Chinese Academy of Sciences, 1 Yanqihu East Road, Beijing, 101408, Beijing, China. · PeopleAI, Inc., 68 Zhichun Road, Beijing, 100098, Beijing, China. +1

May 12, 2026J/K move · Enter open · S save

DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery

Qianxin Xia, Zhiyong Shu, Wenbo Jiang, Jiawei Du+2

University of Electronic Science and Technology of China, Chengdu, China · Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore · Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR), Singapore.

Diffusion-Based Dataset Distillation Dataset Distillation

Open paper AskPDF arXiv

Dataset distillation aims to synthesize a compact proxy dataset that is unreadable or non-raw from the original dataset for privacy protection and highly efficient learning. However, previous approaches typically adopt a single-stage distillation paradigm, which suffers from learning specific patterns that overfit on a prior architecture, consequently suppressing the expression of semantics and leading to performance degradation across heterogeneous architectures. To address this issue, we propose a novel dual-stage distillation framework called

{\textbf{DIVER}}

, which leverages the pre-trained diffusion model to dive deeper into

\textbf{DI}

stilled data

\textbf{V}

\textbf{E}

xpressive semantic

\textbf{R}

ecovery, an entire process of semantic inheritance, guidance, and fusion. Semantic inheritance distills high-level semantics of abstract distilled images into the latent space to filter out architecture-specific ``noise" and retain the intrinsic semantics. Furthermore, semantic guidance improves the preservation of the original semantics by directing the reverse procedure. Finally, semantic fusion is designed to provide semantic guidance only during the concrete phase of the reverse process, preventing semantic ambiguity and artifacts while maintaining the guidance information. Extensive experiments validate the effectiveness and efficiency of DIVER in improving classical distillation techniques and significantly improving cross-architecture generalization, requiring processing time comparable to raw DiT on ImageNet (256

\times

256) with only 4 GB of GPU memory usage. Code is available: https://github.com/einsteinxia/DIVER.

Abstract

Dataset Distillation

Explore similar work

DIVER: Diving Deeper into Distilled Data via Expressive Semantic Recovery

SAS: Semantic-aware Sampling for Generative Dataset Distillation

Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment