Continual Data Unlearning in Diffusion Models via Transition-based Regularization

Sunbeom Jeong; Sehwan Kim; Sangwoo Hong; Jungwoo Lee

cs.LGSep 26, 2026

Continual Data Unlearning in Diffusion Models via Transition-based Regularization

Authors: Sunbeom Jeong, Sehwan Kim, Sangwoo Hong, Jungwoo Lee

Organizations: Seoul National University · Konkuk University

Abstract

Data unlearning in diffusion models aims to remove the influence of specific training examples without suppressing the broader concepts they represent. However, when deletion requests arrive sequentially, updates for new requests can degrade generative utility and undermine earlier deletions. We propose a continual data unlearning framework that uses completed deletion transitions as directional references to regularize future updates. For each request, we record changes in denoiser responses on the same fixed noisy inputs before and after unlearning. Rather than matching full post-deletion responses, we apply a one-sided penalty that discourages reversal along the recorded directions relative to the post-deletion references, while leaving orthogonal response changes and progress beyond these references unpenalized. To keep storage independent of the number of requests, we maintain a fixed-capacity bank of representative transition records. Records are selected based on the local sensitivity of progress along their recorded directions to parameter updates, allowing them to be retained even when their penalties are inactive. Empirical evaluations show that the proposed framework achieves a better balance between deletion persistence and generative utility than existing unlearning baselines as requests accumulate, using only a small transition memory.

Figures & tables

Figure 1: Qualitative comparison of continual data unlearning. Each row shows a target image to be erased and samples generated using its associated prompt. For each method, “Immediate” denotes generation immediately after deleting the target, while “Final” denotes generation after completing the subsequent deletion requests. The prompts used for the generations are provided in Appendix A .

Figure 2: Difference in SSCD for the first unlearned datapoint across sequential deletion requests on CelebA-HQ. Each path tracks a different example from an independent deletion sequence. The x-axis denotes the number of deletion requests processed. The y-axis shows this datapoint’s SSCD after each request minus its SSCD immediately after the first deletion. Positive values indicate that SSCD is higher than it was immediately after the first deletion.

	CelebA-HQ			CIFAR-10
Method	SSCD $\downarrow$	FID $\downarrow$	NLL $\uparrow$	SSCD $\downarrow$	FID $\downarrow$
Naive	$0.8595\,{\scriptstyle\pm 0.0017}$	$20.41\,{\scriptstyle\pm 4.84}$	$1.3281\,{\scriptstyle\pm 0.0296}$	$0.5436\,{\scriptstyle\pm 0.0099}$	$11.38\,{\scriptstyle\pm 1.02}$
NegGrad	$0.1131\,{\scriptstyle\pm 0.0153}$	$441.47\,{\scriptstyle\pm 95.44}$	$18.4781\,{\scriptstyle\pm 1.0963}$	$0.0539\,{\scriptstyle\pm 0.0507}$	$431.07\,{\scriptstyle\pm 64.01}$
EraseDiff	$0.2008\,{\scriptstyle\pm 0.0123}$	$141.31\,{\scriptstyle\pm 87.25}$	$5.8061\,{\scriptstyle\pm 0.1735}$	$0.4567\,{\scriptstyle\pm 0.0110}$	$33.72\,{\scriptstyle\pm 23.26}$
SISS	$0.6781\,{\scriptstyle\pm 0.0267}$	$38.80\,{\scriptstyle\pm 31.13}$	$1.3819\,{\scriptstyle\pm 0.0301}$	$0.4893\,{\scriptstyle\pm 0.0135}$	$\mathbf{11.16}\,{\scriptstyle\pm 2.73}$
ReTrack	$0.4110\,{\scriptstyle\pm 0.0686}$	$31.48\,{\scriptstyle\pm 9.53}$	$2.2861\,{\scriptstyle\pm 0.1426}$	$0.4539\,{\scriptstyle\pm 0.0240}$	$16.80\,{\scriptstyle\pm 2.56}$

Table 1: Results after sequentially unlearning 50 datapoints on CelebA-HQ and CIFAR-10. We report mean and standard deviation over 3 runs with randomly sampled target sets and deletion orders.

	Unlearning performance				General utility (COCO)
Method	Success (%) $\uparrow$	CLIP-IQA $\uparrow$	SSCD $\downarrow$	CLIP score $\uparrow$	CLIP-IQA $\uparrow$	CLIP score $\uparrow$
Pretrained	$0.00$	$0.8112$	$0.6548$	$28.6885$	$0.8866$	$26.6049$
Naive	$25.69\,{\scriptstyle\pm 15.91}$	$0.7169\,{\scriptstyle\pm 0.0365}$	$0.2521\,{\scriptstyle\pm 0.0133}$	$32.0518\,{\scriptstyle\pm 0.2843}$	$0.8871\,{\scriptstyle\pm 0.0031}$	$27.0232\,{\scriptstyle\pm 0.0699}$
NegGrad	$0.00\,{\scriptstyle\pm 0.00}$	$0.1870\,{\scriptstyle\pm 0.0484}$	$0.0207\,{\scriptstyle\pm 0.0074}$	$18.2460\,{\scriptstyle\pm 0.5542}$	$0.2027\,{\scriptstyle\pm 0.0291}$	$14.0139\,{\scriptstyle\pm 0.3815}$
EraseDiff	$43.75\,{\scriptstyle\pm 5.51}$	$0.5312\,{\scriptstyle\pm 0.0492}$	$0.1226\,{\scriptstyle\pm 0.0224}$	$27.4282\,{\scriptstyle\pm 1.1531}$	$0.8887\,{\scriptstyle\pm 0.0081}$	$26.8076\,{\scriptstyle\pm 0.0649}$
SISS	$87.50\,{\scriptstyle\pm 4.17}$	$0.6728\,{\scriptstyle\pm 0.0505}$	$\mathbf{0.1454}\,{\scriptstyle\pm 0.0106}$	$26.1410\,{\scriptstyle\pm 0.6199}$	$0.8907\,{\scriptstyle\pm 0.0059}$	$\mathbf{26.8203}\,{\scriptstyle\pm 0.1474}$

Table 2: Results after sequentially unlearning 48 memorized datapoints from Stable Diffusion v1.4. We evaluate unlearning performance using prompts corresponding to the deletion targets and general generation utility using COCO captions. We report the mean and standard deviation over three runs with different deletion orders.

	SSCD $\downarrow$			FID $\downarrow$
Method	20 deletions	50 deletions	100 deletions	100 deletions
SISS	$0.6936\,{\scriptstyle\pm\,0.0312}$	$0.6826\,{\scriptstyle\pm\,0.0209}$	$0.6873\,{\scriptstyle\pm\,0.0103}$	$\mathbf{17.70}\,{\scriptstyle\pm\,2.02}$
ReTrack	$0.5493\,{\scriptstyle\pm\,0.0354}$	$0.4792\,{\scriptstyle\pm\,0.0395}$	$0.5030\,{\scriptstyle\pm\,0.0258}$	$27.35\,{\scriptstyle\pm\,4.50}$
Ours ( $K_{\mathrm{mem}}=2$ )	$\mathbf{0.4900}\,{\scriptstyle\pm\,0.0198}$	$0.4497\,{\scriptstyle\pm\,0.0528}$	$0.4846\,{\scriptstyle\pm\,0.0254}$	$25.98\,{\scriptstyle\pm\,2.13}$
Ours ( $K_{\mathrm{mem}}=3$ )	$0.4921\,{\scriptstyle\pm\,0.0236}$	$0.4465\,{\scriptstyle\pm\,0.0370}$	$0.4778\,{\scriptstyle\pm\,0.0302}$	$22.18\,{\scriptstyle\pm\,1.71}$
Ours ( $K_{\mathrm{mem}}=4$ )	$0.4913\,{\scriptstyle\pm\,0.0236}$	$\mathbf{0.4434}\,{\scriptstyle\pm\,0.0377}$	$\mathbf{0.4654}\,{\scriptstyle\pm\,0.0347}$	$22.73\,{\scriptstyle\pm\,1.30}$

Table 3: Continual unlearning performance on CelebA-HQ across different deletion sequence lengths and memory capacities. We report SSCD after 20, 50, and 100 deletions and FID after 100 deletions, with

K_{\mathrm{mem}}\in\{2,3,4\}

for our method. Results are the mean and standard deviation over three runs with different target sets and deletion orders.

	CelebA-HQ		Stable Diffusion v1.4
Penalty	SSCD $\downarrow$	FID $\downarrow$	Success (%) $\uparrow$	CLIP-IQA $\uparrow$	SSCD $\downarrow$	CLIP score $\uparrow$
Full-output	$0.4326\,{\scriptstyle\pm 0.0404}$	$33.69\,{\scriptstyle\pm 9.26}$	$92.36\,{\scriptstyle\pm 1.20}$	$0.8013\,{\scriptstyle\pm 0.0309}$	$0.1155\,{\scriptstyle\pm 0.0168}$	$\mathbf{26.6624}\,{\scriptstyle\pm 0.2519}$
Symmetric	$0.4391\,{\scriptstyle\pm 0.0279}$	$30.03\,{\scriptstyle\pm 10.79}$	$93.75\,{\scriptstyle\pm 3.61}$	$0.7937\,{\scriptstyle\pm 0.0299}$	$\mathbf{0.1061}\,{\scriptstyle\pm 0.0190}$	$26.4982\,{\scriptstyle\pm 0.1673}$
Ours	$\mathbf{0.3975}\,{\scriptstyle\pm 0.0680}$	$\mathbf{27.62}\,{\scriptstyle\pm 9.03}$	$\mathbf{98.61}\,{\scriptstyle\pm 1.20}$	$\mathbf{0.8052}\,{\scriptstyle\pm 0.0349}$	$0.1485\,{\scriptstyle\pm 0.0189}$	$26.5527\,{\scriptstyle\pm 0.3633}$

Table 4: Comparison of response-preservation penalties on CelebA-HQ and Stable Diffusion v1.4 after 50 and 48 deletions, respectively. All variants use

K_{\mathrm{mem}}=2

. We report the mean and standard deviation over three runs with different deletion orders.

	CelebA-HQ		Stable Diffusion v1.4
Selection	SSCD $\downarrow$	FID $\downarrow$	Success (%) $\uparrow$	CLIP-IQA $\uparrow$	SSCD $\downarrow$	CLIP $\uparrow$
FIFO	$0.4400\,{\scriptstyle\pm 0.0269}$	$32.78\,{\scriptstyle\pm 10.48}$	$93.06\,{\scriptstyle\pm 3.18}$	$0.7907\,{\scriptstyle\pm 0.0293}$	$\mathbf{0.1205}\,{\scriptstyle\pm 0.0150}$	$26.2898\,{\scriptstyle\pm 1.0175}$
Random	$0.4379\,{\scriptstyle\pm 0.0288}$	$31.39\,{\scriptstyle\pm 9.26}$	$93.75\,{\scriptstyle\pm 5.51}$	$0.7916\,{\scriptstyle\pm 0.0458}$	$0.1216\,{\scriptstyle\pm 0.0111}$	$\mathbf{27.1066}\,{\scriptstyle\pm 0.4785}$
Ours	$\mathbf{0.3975}\,{\scriptstyle\pm 0.0680}$	$\mathbf{27.62}\,{\scriptstyle\pm 9.03}$	$\mathbf{98.61}\,{\scriptstyle\pm 1.20}$	$\mathbf{0.8052}\,{\scriptstyle\pm 0.0349}$	$0.1485\,{\scriptstyle\pm 0.0189}$	$26.5527\,{\scriptstyle\pm 0.3633}$

Table 5: Comparison of memory-selection rules on CelebA-HQ and Stable Diffusion v1.4 after 50 and 48 deletions, respectively. All variants use

K_{\mathrm{mem}}=2

. We report the mean and standard deviation over three runs with different deletion orders.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	CIFAR-10	CelebA-HQ	Stable Diffusion v1.4
Updates per request	$60$	$40$	$35$
Learning rate	$2\times 10^{-5}$	$5\times 10^{-6}$	$10^{-5}$
AdamW $(\beta_{1},\beta_{2})$	$(0.95,0.999)$	$(0.95,0.999)$	$(0.9,0.999)$
Weight decay	$10^{-6}$	$10^{-6}$	$10^{-2}$
AdamW $\epsilon$	$10^{-8}$	$10^{-8}$	$10^{-8}$
Probes $Q$	$4$	$4$	$4$

Appendix

Table 6: Training and correction hyperparameters.

K_{\mathrm{mem}}

counts older records and excludes the separately stored newest record. Memory-capacity and penalty variants override the corresponding entries.

	CelebA-HQ		Stable Diffusion v1.4
Method	SSCD $\downarrow$	FID $\downarrow$	Success (%) $\uparrow$	CLIP-IQA $\uparrow$	SSCD $\downarrow$	CLIP score $\uparrow$
SISS	$0.6985\,{\scriptstyle\pm 0.0256}$	$\mathbf{25.06}\,{\scriptstyle\pm 4.78}$	$74.67\,{\scriptstyle\pm 14.64}$	$0.6782\,{\scriptstyle\pm 0.0574}$	$0.1611\,{\scriptstyle\pm 0.0218}$	$26.2896\,{\scriptstyle\pm 0.8763}$
ReTrack	$0.4292\,{\scriptstyle\pm 0.0228}$	$29.01\,{\scriptstyle\pm 2.43}$	$85.33\,{\scriptstyle\pm 9.31}$	$0.7942\,{\scriptstyle\pm 0.0346}$	$0.2286\,{\scriptstyle\pm 0.0163}$	$26.7059\,{\scriptstyle\pm 0.7214}$
Ours	$\mathbf{0.4115}\,{\scriptstyle\pm 0.0187}$	$28.90\,{\scriptstyle\pm 3.24}$	$\mathbf{90.67}\,{\scriptstyle\pm 2.79}$	$\mathbf{0.7969}\,{\scriptstyle\pm 0.0171}$	$\mathbf{0.1545}\,{\scriptstyle\pm 0.0250}$	$\mathbf{26.7712}\,{\scriptstyle\pm 0.5602}$

Appendix

Table 7: Sensitivity to request order on CelebA-HQ and Stable Diffusion v1.4. For each setting, we evaluate five different deletion orders using a fixed target set: 50 targets for CelebA-HQ and 30 targets for Stable Diffusion. We report the mean and standard deviation across these orders. Our method uses

K_{\mathrm{mem}}=2

in both settings.

All evaluated targets
	SSCD $\downarrow$
Method	Immediately after deletion	After 50 deletions	Mean rebound
SISS	$0.6069\,{\scriptstyle\pm 0.0044}$	$0.6814\,{\scriptstyle\pm 0.0093}$	$\mathbf{0.0746}\,{\scriptstyle\pm 0.0136}$
ReTrack	$0.3631\,{\scriptstyle\pm 0.0237}$	$0.4810\,{\scriptstyle\pm 0.0091}$	$0.1210\,{\scriptstyle\pm 0.0151}$
Ours	$\mathbf{0.3529}\,{\scriptstyle\pm 0.0153}$	$\mathbf{0.4408}\,{\scriptstyle\pm 0.0326}$	$0.1001\,{\scriptstyle\pm 0.0325}$
Common targets with low post-deletion SSCD

Appendix

Table 8: SSCD rebound on CelebA-HQ over three runs of 50 deletions. Ours uses

K_{\mathrm{mem}}=4

. The lower panel requires immediate SSCD

\leq 0.5

under all methods.

Method	Time/request (s)	Relative time	Training peak (MiB)
ReTrack	$224.47\pm 19.38$	$1.00\times$	$21317.48\pm 0.00$
Ours ( $K_{\mathrm{mem}}=2$ )	$277.42\pm 16.23$	$1.24\times$	$21331.51\pm 1.16$
Ours ( $K_{\mathrm{mem}}=3$ )	$272.15\pm 6.97$	$1.22\times$	$21331.90\pm 0.67$
Ours ( $K_{\mathrm{mem}}=4$ )	$285.41\pm 5.37$	$1.28\times$	$21332.28\pm 0.67$

Appendix

Table 9: CelebA-HQ resource costs across three deletion sequences.

Diffusion Model Unlearning

Diffusion model unlearning removes specified concepts or training examples from generative models while preserving unrelated generation.

See all 77 papers

Oct 20, 2025

Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning

Data unlearning aims to remove the influence of specific training samples from a trained model. In fine-tuning methods, data unlearning relies primarily on loss maximization over forget samples, which often leads to quality degradation or incomplete forgetting. Existing methods perform unlearning uniformly across diffusion stages, ignoring diffusion dynamics from noise to data. Our systematic study of diffusion phases shows that forgetting in diffusion models is uneven across time and frequency, with theoretical justification of distributive distortion and forgetting-utility trade-off. By selectively forgetting time and frequency in diffusion models, we achieve both higher unlearning success rates and improved generation quality across diverse settings, including both conditional and unconditional scenarios. We also introduce an improved SSCD metric that measures dissimilarity using a normalized perturbation distance. Together, we provide practical insights for understanding and improving data unlearning in diffusion models.

Jinseong Park, Mijung Park

Korea Institute for Advanced Study · University of British Columbia

Diffusion Model Unlearning Machine Unlearning Diffusion Models

Ask PDF

May 29, 2026

Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints

Unlearning in diffusion models aims to remove undesirable data or concepts while preserving the utility of pretrained models -- two fundamentally conflicting objectives. We propose a principled constrained optimization framework that formulates unlearning as minimizing the deviation from a pretrained model, subject to explicit separation constraints from the unlearning distributions. Specifically, we formulate three constrained optimization problems based on reverse and forward KL divergences, and likelihood constraints. The first two generalize existing approaches for concept and data unlearning, while the third offers a novel and natural formulation for unlearning. Despite the nonconvexity of the KL constraints, we establish strong duality for all three problems, enabling us to explicitly characterize their optimal solutions as unlearning targets and develop primal-dual algorithms for each formulation. Experimental results demonstrate that our KL-constrained approach achieves superior retention-unlearning tradeoffs compared to weight-based baselines for concept and data unlearning, and that our likelihood-based approach matches unlearning effectiveness while better preserving retained concepts compared to baselines.

Shervin Khalafi, Alejandro Ribeiro, Dongsheng Ding

University of Pennsylvania · University of Tennessee, Knoxville

Primal-Dual Optimization Diffusion Model Unlearning Constrained Optimization Machine Unlearning Diffusion Models

Ask PDF

Sep 21, 2026

Mitigating Sequential Reappearance in Diffusion Data-Point Unlearning

Diffusion data-point unlearning is typically evaluated immediately after each deletion, even though subsequent requests may repeatedly update the same model. We identify sequential reappearance, a failure mode in which an instance that is initially judged to be forgotten later returns to the memorized regime without reuse of the deleted data or adversarial fine-tuning. To capture this behavior, we introduce a target-level evaluation protocol that tracks whether each target is forgotten immediately, remains forgotten at the end of the sequence, or reappears during subsequent deletions. We further find that targets that later reappear exhibit sharper local denoising-loss geometry after deletion than targets that remain forgotten.

Donghyun Kim, Taehyuk Lee, Jinyeong Kim +4

Department of Computer Science and Engineering, Konkuk University Trustworthy Machine Learning Lab Seoul, Republic of Korea

Memorization in Generative Models Diffusion Model Unlearning Machine Unlearning Machine Unlearning Evaluation

Ask PDF

Abstract

Diffusion Model Unlearning

Explore similar work

Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning

Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints

Mitigating Sequential Reappearance in Diffusion Data-Point Unlearning