Dataset Distillation

Dataset distillation aims to replace a large training set with a compact set of examples that lets models learn effectively while reducing storage, communication, or training costs. Common approaches match gradients or training trajectories, align feature or data distributions, select representative prototypes, or generate examples with diffusion models; newer work also targets text, graphs, time series, video, and multimodal data. Researchers additionally study how to compress labels and bits, and how to preserve robustness, fairness, and performance across model architectures. A central open tension is whether distilled sets reliably outperform simpler random subsets or coresets, especially at large scale, while balancing downstream accuracy against faithful coverage of the original data.