cs.LGSep 28, 2026

Data Unlearning via Inverse Distillation

Authors: Aleksei Leonov, Nikita Kornilov, Zhenhe Zhang, Evgeny Burnaev, Iaroslav Koshelev, Alexander Korotin

Organizations: AI Foundation lab, Moscow, Russia · MIRAI, Moscow, Russia · Applied AI Institute, Moscow, Russia · BRAIn Lab, Moscow, Russia · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-step matching models, including flow and diffusion models, produce high-quality outputs but incur substantial inference costs and may reproduce unwanted components of their training datasets. We introduce Inverse Distillation Unlearning (IDU), a unified framework that simultaneously distills a teacher multi-step matching model into an efficient one-step student generator and suppresses outputs corresponding to a designated training subset. We first formulate distillation as a min-max objective over a data distribution and then represent this distribution as a mixture of the forget-set and the generated distributions. This allows us to compare this mixture with the teacher's training distribution and recover only the retained data at the optimum. Our method requires only a pretrained full-data teacher and data from the forget set, without access to retained training examples, extra feature extractors or classifiers. Extensive experiments on MNIST and CIFAR-10 datasets under flow-matching and score-based diffusion settings demonstrate that IDU substantially reduces the generation frequency of forgotten classes while preserving generation quality on the retained classes. To the best of our knowledge, IDU is the first unified framework for simultaneous unlearning and distillation in unconditional flow-matching and score-based models.

Figures & tables

Figure 1: Pipeline of our IDU framework . Forget and generated samples form two forward-noising branches. The framework alternates between two steps: first, update the fake model on both branches with weights ρ\rho and 1−ρ1-\rho to detect generations similar to the forget samples, then update the generator to suppress these generations using the frozen teacher and the fake model.
 1B∑i=1B[ρ∥fψ,ti(xti,iF)−sti,iF∥2+(1−ρ)∥fψ,ti(xti,iθ)−sti,iθ∥2];\frac{1}{B}\sum\limits^{B}_{i=1}\left[\rho\|f_{\psi,t_{i}}(x^{F}_{t_{i},i})-s^{F}_{t_{i},i}\|^{2}+(1-\rho)\|f_{\psi,t_{i}}(x^{\theta}_{t_{i},i})-s^{\theta}_{t_{i},i}\|^{2}\right]; 
Algorithm 1 Inverse Distillation Unlearning
 MNISTCIFAR-10
ModeFID ↓\downarrowFGR (%) ↓\downarrow Class 3 / Class 7FID ↓\downarrowFGR (%) ↓\downarrow Class 1 / Class 9
FM
Pretrain0.88±0.010.88\pm 0.0110.40±0.1210.40\pm 0.12 10.10±0.2610.10\pm 0.263.66±0.033.66\pm 0.0312.50±0.0812.50\pm 0.08 11.18±0.1211.18\pm 0.12
Pure distillation3.23±0.033.23\pm 0.0310.50±0.1510.50\pm 0.15 10.53±0.0910.53\pm 0.094.35±0.054.35\pm 0.057.58±0.097.58\pm 0.09 8.75±0.218.75\pm 0.21
Forgotten classes{3,7}\{3,7\}{1,9}\{1,9\}
Retrain teacher0.340±0.0010.340\pm 0.001—3.93±0.023.93\pm 0.02—
Table 1: Main FM/SiD results on MNIST and CIFAR-10. FID uses full-data references for Pretrain/Pure distillation and retained-data references for Retrain/IDU; FGR is reported per forgotten class. Values are mean ±\pm standard deviation over five runs, except †\dagger values from ( Zhou et al., 2024 ) .
 FM MNISTFM CIFAR-10SiD CIFAR-10
Forgotten ClassRetain FID ↓\downarrowFGR (%) ↓\downarrowRetain FID ↓\downarrowFGR (%) ↓\downarrowRetain FID ↓\downarrowFGR (%) ↓\downarrow
05.05±0.035.05\pm 0.030.10±0.010.10\pm 0.016.52±0.066.52\pm 0.060.76±0.030.76\pm 0.033.54±0.023.54\pm 0.020.58±0.030.58\pm 0.03
13.46±0.043.46\pm 0.040.03±0.010.03\pm 0.016.45±0.096.45\pm 0.091.06±0.071.06\pm 0.072.96±0.012.96\pm 0.010.61±0.030.61\pm 0.03
23.84±0.023.84\pm 0.020.12±0.010.12\pm 0.017.83±0.057.83\pm 0.050.73±0.040.73\pm 0.044.07±0.054.07\pm 0.050.87±0.040.87\pm 0.04
35.23±0.035.23\pm 0.030.10±0.010.10\pm 0.017.82±0.087.82\pm 0.081.51±0.041.51\pm 0.044.10±0.024.10\pm 0.021.11±0.021.11\pm 0.02
44.71±0.034.71\pm 0.030.13±0.020.13\pm 0.028.53±0.088.53\pm 0.080.82±0.040.82\pm 0.044.13±0.034.13\pm 0.030.78±0.050.78\pm 0.05
Table 2: Single-class robustness of IDU with FM on MNIST and CIFAR-10 and with SiD on CIFAR-10. The final row averages the per-class means across the ten forget-class experiments.
Flow MatchingSiD
ρ\rhoFID ↓\downarrowFGR 1 (%) ↓\downarrowFGR 9 (%) ↓\downarrowρ\rhoFID ↓\downarrowFGR 1 (%) ↓\downarrowFGR 9 (%) ↓\downarrow
0.050.056.72±0.076.72\pm 0.073.04±0.083.04\pm 0.085.02±0.085.02\pm 0.08—
0.10.16.21±0.056.21\pm 0.053.75±0.093.75\pm 0.093.80±0.083.80\pm 0.08—
0.20.25.64±0.045.64\pm 0.043.35±0.033.35\pm 0.033.04±0.103.04\pm 0.100.20.23.15±0.03\mathbf{3.15\pm 0.03}1.11±0.06\mathbf{1.11\pm 0.06}1.25±0.06\mathbf{1.25\pm 0.06}
0.40.45.87±0.075.87\pm 0.070.62±0.060.62\pm 0.060.48±0.020.48\pm 0.020.40.43.53±0.043.53\pm 0.040.74±0.050.74\pm 0.050.60±0.010.60\pm 0.01
0.60.65.81±0.05\mathbf{5.81\pm 0.05}0.56±0.05\mathbf{0.56\pm 0.05}0.39±0.03\mathbf{0.39\pm 0.03}0.60.64.68±0.064.68\pm 0.061.16±0.051.16\pm 0.050.76±0.020.76\pm 0.02
Table 3: Effect of the forget-mixture weight ρ\rho on CIFAR-10 for FM- and SiD-based IDU when jointly forgetting automobile (class 1) and truck (class 9). The selected configuration for each setup is bold; dashes denote settings not evaluated with SiD. FM training at ρ=0.9\rho=0.9 diverges immediately.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

 FMSiD
ModeFID ↓\downarrowFGR (%) ↓\downarrow Class 1 / Class 9FID ↓\downarrowFGR (%) ↓\downarrow Class 1 / Class 9
Pretrain3.66±0.033.66\pm 0.0312.50±0.0812.50\pm 0.08 11.18±0.1211.18\pm 0.121.97†1.97^{\dagger}11.13±0.1211.13\pm 0.12 10.04±0.1510.04\pm 0.15
Pure distillation4.35±0.054.35\pm 0.057.58±0.097.58\pm 0.09 8.75±0.218.75\pm 0.211.92±0.02 †1.92\pm 0.02\,^{\dagger}10.11±0.1410.11\pm 0.14 10.66±0.1410.66\pm 0.14
Forgotten classes{1,9}\{1,9\}{1,9}\{1,9\}
IDU from scratch (ρCIFAR-10=0.6)(\rho_{\mathrm{CIFAR\text{-}10}}=0.6) (ρSiD=0.2)(\rho_{\mathrm{SiD}}=0.2)5.81±0.055.81\pm 0.050.56±0.050.56\pm 0.05 0.39±0.030.39\pm 0.033.15±0.033.15\pm 0.031.11±0.061.11\pm 0.06 1.25±0.061.25\pm 0.06
IDU Fine-tuning (ρCIFAR-10=0.6)(\rho_{\mathrm{CIFAR\text{-}10}}=0.6) (ρSiD=0.2)(\rho_{\mathrm{SiD}}=0.2)5.17±0.085.17\pm 0.080.89±0.020.89\pm 0.02 0.42±0.030.42\pm 0.033.12±0.063.12\pm 0.062.69±0.062.69\pm 0.06 2.60±0.042.60\pm 0.04
Appendix
Table 4: Fine-tune FM/SiD results for the purely distilled generators on CIFAR-10. FID uses full-data references for Pretrain/Pure distillation and retained-data references for IDU and fine-tuning; FGR is reported per forgotten class. Values are mean ±\pm standard deviation over five runs, except †\dagger values from ( Zhou et al., 2024 ) .
ρ\rhoRetain FID ↓\downarrowFGR 3 (%) ↓\downarrowFGR 7 (%) ↓\downarrowAdditional suppressed digits
0.050.054.29±0.024.29\pm 0.027.07±0.067.07\pm 0.068.18±0.068.18\pm 0.06—
0.10.13.96±0.043.96\pm 0.043.87±0.093.87\pm 0.094.62±0.104.62\pm 0.10—
0.20.24.81±0.084.81\pm 0.082.22±0.072.22\pm 0.071.63±0.071.63\pm 0.07—
0.4\mathbf{0.4}3.57±0.03\mathbf{3.57\pm 0.03}0.16±0.01\mathbf{0.16\pm 0.01}0.16±0.01\mathbf{0.16\pm 0.01}—
0.60.611.20±0.1011.20\pm 0.1000002, 5, 8
0.80.810.58±0.0710.58\pm 0.0700002, 5
Appendix
Table 5: Effect of the forget-mixture weight ρ\rho on FM-based IDU for MNIST when jointly forgetting digits 3 and 7. The main configuration is bold. The final column lists qualitatively suppressed digits in addition to the target digits 3 and 7.
Figure 2: FM on MNIST: teacher and IDU samples when forgetting digits 3 and 7.
Figure 3: SiD on MNIST: teacher and IDU samples when forgetting digits 3 and 7.
Figure 4: FM on CIFAR-10: teacher and IDU samples when forgetting automobile (class 1) and truck (class 9).
Figure 5: SiD on CIFAR-10: teacher and IDU samples when forgetting automobile (class 1) and truck (class 9).
More in this topic

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 343 papers

Explore similar work

CardsList
Oct 6, 2026

Consistent Distribution Matching for Data-Free Diffusion Distillation

Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only two models, a frozen teacher and a trainable student, and optimizes one objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256×\times256, our method attains an FID of 2.04 with a single function evaluation (1-NFE) and a 4-NFE FID of 1.37 within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data. Our code code and model are available at https://consistentdmd.github.io/.

Yuxiang Fu, Qi Yan, Zike Wu +4

University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair

Diffusion Model DistillationDiffusion Model Inference AccelerationDistribution Matching DistillationData-Free Knowledge Distillation
AskPDF
Mar 17, 2026

Unlearning for One-Step Generative Models via Unbalanced Optimal Transport

Recent advances in one-step generative frameworks, such as flow map models, have significantly improved the efficiency of image generation by learning direct noise-to-data mappings in a single forward pass. However, machine unlearning for ensuring the safety of these powerful generators remains entirely unexplored. Existing diffusion unlearning methods are inherently incompatible with these one-step models, as they rely on a multi-step iterative denoising process. In this work, we propose UOT-Unlearn, a novel plug-and-play class unlearning framework for one-step generative models based on the Unbalanced Optimal Transport (UOT). Our method formulates unlearning as a principled trade-off between a forget cost, which suppresses the target class, and an ff-divergence penalty, which preserves overall generation fidelity via relaxed marginal constraints. By leveraging UOT, our method enables the probability mass of the forgotten class to be smoothly redistributed to the remaining classes, rather than collapsing into low-quality or noise-like samples. Experimental results on CIFAR-10 and ImageNet-256 demonstrate that our framework achieves superior unlearning success (PUL) and retention quality (u-FID), significantly outperforming baselines.

Hyundo Choi, Junhyeong An, Jinseong Park +1

Sungkyunkwan University, Seoul, Republic of Korea · Korea Institute for Advanced Study, Seoul, Republic of Korea

Machine Unlearning
AskPDF
Oct 20, 2025

Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning

Data unlearning aims to remove the influence of specific training samples from a trained model. In fine-tuning methods, data unlearning relies primarily on loss maximization over forget samples, which often leads to quality degradation or incomplete forgetting. Existing methods perform unlearning uniformly across diffusion stages, ignoring diffusion dynamics from noise to data. Our systematic study of diffusion phases shows that forgetting in diffusion models is uneven across time and frequency, with theoretical justification of distributive distortion and forgetting-utility trade-off. By selectively forgetting time and frequency in diffusion models, we achieve both higher unlearning success rates and improved generation quality across diverse settings, including both conditional and unconditional scenarios. We also introduce an improved SSCD metric that measures dissimilarity using a normalized perturbation distance. Together, we provide practical insights for understanding and improving data unlearning in diffusion models.

Jinseong Park, Mijung Park

Korea Institute for Advanced Study · University of British Columbia

Diffusion Model UnlearningMachine UnlearningDiffusion Models
AskPDF