Data Unlearning via Inverse Distillation

Aleksei Leonov; Nikita Kornilov; Zhenhe Zhang; Evgeny Burnaev; Iaroslav Koshelev; Alexander Korotin

cs.LGSep 28, 2026

Data Unlearning via Inverse Distillation

Authors: Aleksei Leonov, Nikita Kornilov, Zhenhe Zhang, Evgeny Burnaev, Iaroslav Koshelev, Alexander Korotin

Organizations: AI Foundation lab, Moscow, Russia · MIRAI, Moscow, Russia · Applied AI Institute, Moscow, Russia · BRAIn Lab, Moscow, Russia ·

Abstract

Multi-step matching models, including flow and diffusion models, produce high-quality outputs but incur substantial inference costs and may reproduce unwanted components of their training datasets. We introduce Inverse Distillation Unlearning (IDU), a unified framework that simultaneously distills a teacher multi-step matching model into an efficient one-step student generator and suppresses outputs corresponding to a designated training subset. We first formulate distillation as a min-max objective over a data distribution and then represent this distribution as a mixture of the forget-set and the generated distributions. This allows us to compare this mixture with the teacher's training distribution and recover only the retained data at the optimum. Our method requires only a pretrained full-data teacher and data from the forget set, without access to retained training examples, extra feature extractors or classifiers. Extensive experiments on MNIST and CIFAR-10 datasets under flow-matching and score-based diffusion settings demonstrate that IDU substantially reduces the generation frequency of forgotten classes while preserving generation quality on the retained classes. To the best of our knowledge, IDU is the first unified framework for simultaneous unlearning and distillation in unconditional flow-matching and score-based models.

Figures & tables

Figure 1: Pipeline of our IDU framework . Forget and generated samples form two forward-noising branches. The framework alternates between two steps: first, update the fake model on both branches with weights $\rho$ and $1-\rho$ to detect generations similar to the forget samples, then update the generator to suppress these generations using the frozen teacher and the fake model.

\frac{1}{B}\sum\limits^{B}_{i=1}\left[\rho\|f_{\psi,t_{i}}(x^{F}_{t_{i},i})-s^{F}_{t_{i},i}\|^{2}+(1-\rho)\|f_{\psi,t_{i}}(x^{\theta}_{t_{i},i})-s^{\theta}_{t_{i},i}\|^{2}\right];

Algorithm 1 Inverse Distillation Unlearning

	MNIST		CIFAR-10
Mode	FID $\downarrow$	FGR (%) $\downarrow$ Class 3 / Class 7	FID $\downarrow$	FGR (%) $\downarrow$ Class 1 / Class 9
FM
Pretrain	$0.88\pm 0.01$	$10.40\pm 0.12$ $10.10\pm 0.26$	$3.66\pm 0.03$	$12.50\pm 0.08$ $11.18\pm 0.12$
Pure distillation	$3.23\pm 0.03$	$10.50\pm 0.15$ $10.53\pm 0.09$	$4.35\pm 0.05$	$7.58\pm 0.09$ $8.75\pm 0.21$
Forgotten classes	$\{3,7\}$		$\{1,9\}$
Retrain teacher	$0.340\pm 0.001$	—	$3.93\pm 0.02$	—

Table 1: Main FM/SiD results on MNIST and CIFAR-10. FID uses full-data references for Pretrain/Pure distillation and retained-data references for Retrain/IDU; FGR is reported per forgotten class. Values are mean

\pm

standard deviation over five runs, except

\dagger

values from ( Zhou et al., 2024 ) .

	FM MNIST		FM CIFAR-10		SiD CIFAR-10
Forgotten Class	Retain FID $\downarrow$	FGR (%) $\downarrow$	Retain FID $\downarrow$	FGR (%) $\downarrow$	Retain FID $\downarrow$	FGR (%) $\downarrow$
0	$5.05\pm 0.03$	$0.10\pm 0.01$	$6.52\pm 0.06$	$0.76\pm 0.03$	$3.54\pm 0.02$	$0.58\pm 0.03$
1	$3.46\pm 0.04$	$0.03\pm 0.01$	$6.45\pm 0.09$	$1.06\pm 0.07$	$2.96\pm 0.01$	$0.61\pm 0.03$
2	$3.84\pm 0.02$	$0.12\pm 0.01$	$7.83\pm 0.05$	$0.73\pm 0.04$	$4.07\pm 0.05$	$0.87\pm 0.04$
3	$5.23\pm 0.03$	$0.10\pm 0.01$	$7.82\pm 0.08$	$1.51\pm 0.04$	$4.10\pm 0.02$	$1.11\pm 0.02$
4	$4.71\pm 0.03$	$0.13\pm 0.02$	$8.53\pm 0.08$	$0.82\pm 0.04$	$4.13\pm 0.03$	$0.78\pm 0.05$

Table 2: Single-class robustness of IDU with FM on MNIST and CIFAR-10 and with SiD on CIFAR-10. The final row averages the per-class means across the ten forget-class experiments.

Flow Matching				SiD
$\rho$	FID $\downarrow$	FGR 1 (%) $\downarrow$	FGR 9 (%) $\downarrow$	$\rho$	FID $\downarrow$	FGR 1 (%) $\downarrow$	FGR 9 (%) $\downarrow$
$0.05$	$6.72\pm 0.07$	$3.04\pm 0.08$	$5.02\pm 0.08$	—
$0.1$	$6.21\pm 0.05$	$3.75\pm 0.09$	$3.80\pm 0.08$	—
$0.2$	$5.64\pm 0.04$	$3.35\pm 0.03$	$3.04\pm 0.10$	$0.2$	$\mathbf{3.15\pm 0.03}$	$\mathbf{1.11\pm 0.06}$	$\mathbf{1.25\pm 0.06}$
$0.4$	$5.87\pm 0.07$	$0.62\pm 0.06$	$0.48\pm 0.02$	$0.4$	$3.53\pm 0.04$	$0.74\pm 0.05$	$0.60\pm 0.01$
$0.6$	$\mathbf{5.81\pm 0.05}$	$\mathbf{0.56\pm 0.05}$	$\mathbf{0.39\pm 0.03}$	$0.6$	$4.68\pm 0.06$	$1.16\pm 0.05$	$0.76\pm 0.02$

Table 3: Effect of the forget-mixture weight

\rho

on CIFAR-10 for FM- and SiD-based IDU when jointly forgetting automobile (class 1) and truck (class 9). The selected configuration for each setup is bold; dashes denote settings not evaluated with SiD. FM training at

\rho=0.9

diverges immediately.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	FM		SiD
Mode	FID $\downarrow$	FGR (%) $\downarrow$ Class 1 / Class 9	FID $\downarrow$	FGR (%) $\downarrow$ Class 1 / Class 9
Pretrain	$3.66\pm 0.03$	$12.50\pm 0.08$ $11.18\pm 0.12$	$1.97^{\dagger}$	$11.13\pm 0.12$ $10.04\pm 0.15$
Pure distillation	$4.35\pm 0.05$	$7.58\pm 0.09$ $8.75\pm 0.21$	$1.92\pm 0.02\,^{\dagger}$	$10.11\pm 0.14$ $10.66\pm 0.14$
Forgotten classes	$\{1,9\}$		$\{1,9\}$
IDU from scratch $(\rho_{\mathrm{CIFAR\text{-}10}}=0.6)$ $(\rho_{\mathrm{SiD}}=0.2)$	$5.81\pm 0.05$	$0.56\pm 0.05$ $0.39\pm 0.03$	$3.15\pm 0.03$	$1.11\pm 0.06$ $1.25\pm 0.06$
IDU Fine-tuning $(\rho_{\mathrm{CIFAR\text{-}10}}=0.6)$ $(\rho_{\mathrm{SiD}}=0.2)$	$5.17\pm 0.08$	$0.89\pm 0.02$ $0.42\pm 0.03$	$3.12\pm 0.06$	$2.69\pm 0.06$ $2.60\pm 0.04$

Appendix

Table 4: Fine-tune FM/SiD results for the purely distilled generators on CIFAR-10. FID uses full-data references for Pretrain/Pure distillation and retained-data references for IDU and fine-tuning; FGR is reported per forgotten class. Values are mean

\pm

standard deviation over five runs, except

\dagger

values from ( Zhou et al., 2024 ) .

$\rho$	Retain FID $\downarrow$	FGR 3 (%) $\downarrow$	FGR 7 (%) $\downarrow$	Additional suppressed digits
$0.05$	$4.29\pm 0.02$	$7.07\pm 0.06$	$8.18\pm 0.06$	—
$0.1$	$3.96\pm 0.04$	$3.87\pm 0.09$	$4.62\pm 0.10$	—
$0.2$	$4.81\pm 0.08$	$2.22\pm 0.07$	$1.63\pm 0.07$	—
$\mathbf{0.4}$	$\mathbf{3.57\pm 0.03}$	$\mathbf{0.16\pm 0.01}$	$\mathbf{0.16\pm 0.01}$	—
$0.6$	$11.20\pm 0.10$	$0$	$0$	2, 5, 8
$0.8$	$10.58\pm 0.07$	$0$	$0$	2, 5

Appendix

Table 5: Effect of the forget-mixture weight

\rho

on FM-based IDU for MNIST when jointly forgetting digits 3 and 7. The main configuration is bold. The final column lists qualitatively suppressed digits in addition to the target digits 3 and 7.

Panel of Figure 2 — Figure 2: FM on MNIST: teacher and IDU samples when forgetting digits 3 and 7.

Panel of Figure 3 — Figure 3: SiD on MNIST: teacher and IDU samples when forgetting digits 3 and 7.

Panel of Figure 4 — Figure 4: FM on CIFAR-10: teacher and IDU samples when forgetting automobile (class 1) and truck (class 9).

Panel of Figure 5 — Figure 5: SiD on CIFAR-10: teacher and IDU samples when forgetting automobile (class 1) and truck (class 9).

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 343 papers

Consistent Distribution Matching for Data-Free Diffusion Distillation
Oct 6, 2026Yuxiang Fu, Qi Yan, Zike Wu +4Diffusion Model Distillation Diffusion Model Inference Acceleration
University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair
Unlearning for One-Step Generative Models via Unbalanced Optimal Transport
Mar 17, 2026Hyundo Choi, Junhyeong An, Jinseong Park +1Machine Unlearning
Sungkyunkwan University, Seoul, Republic of Korea · Korea Institute for Advanced Study, Seoul, Republic of Korea
Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning
Oct 20, 2025Jinseong Park, Mijung ParkDiffusion Model Unlearning Machine Unlearning
Korea Institute for Advanced Study · University of British Columbia

Oct 6, 2026J/K move · Enter open · S save

Consistent Distribution Matching for Data-Free Diffusion Distillation

Yuxiang Fu, Qi Yan, Zike Wu, Yongxing Zhang+3

University of British Columbia · Vector Institute for AI · Canada CIFAR AI Chair

Diffusion Model Distillation Diffusion Model Inference Acceleration Distribution Matching Distillation Data-Free Knowledge Distillation

Open paper AskPDF arXiv

Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only two models, a frozen teacher and a trainable student, and optimizes one objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256

\times

256, our method attains an FID of 2.04 with a single function evaluation (1-NFE) and a 4-NFE FID of 1.37 within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data. Our code code and model are available at https://consistentdmd.github.io/.

Abstract

Machine Unlearning

Explore similar work

Consistent Distribution Matching for Data-Free Diffusion Distillation

Unlearning for One-Step Generative Models via Unbalanced Optimal Transport

Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning