Continual Data Unlearning in Diffusion Models via Transition-based Regularization

Sunbeom Jeong; Sehwan Kim; Sangwoo Hong; Jungwoo Lee

cs.LGSep 26, 2026

Continual Data Unlearning in Diffusion Models via Transition-based Regularization

Authors: Sunbeom Jeong, Sehwan Kim, Sangwoo Hong, Jungwoo Lee

Organizations: Seoul National University · Konkuk University

Abstract

Data unlearning in diffusion models aims to remove the influence of specific training examples without suppressing the broader concepts they represent. However, when deletion requests arrive sequentially, updates for new requests can degrade generative utility and undermine earlier deletions. We propose a continual data unlearning framework that uses completed deletion transitions as directional references to regularize future updates. For each request, we record changes in denoiser responses on the same fixed noisy inputs before and after unlearning. Rather than matching full post-deletion responses, we apply a one-sided penalty that discourages reversal along the recorded directions relative to the post-deletion references, while leaving orthogonal response changes and progress beyond these references unpenalized. To keep storage independent of the number of requests, we maintain a fixed-capacity bank of representative transition records. Records are selected based on the local sensitivity of progress along their recorded directions to parameter updates, allowing them to be retained even when their penalties are inactive. Empirical evaluations show that the proposed framework achieves a better balance between deletion persistence and generative utility than existing unlearning baselines as requests accumulate, using only a small transition memory.

Figures & tables

Figure 1: Qualitative comparison of continual data unlearning. Each row shows a target image to be erased and samples generated using its associated prompt. For each method, “Immediate” denotes generation immediately after deleting the target, while “Final” denotes generation after completing the subsequent deletion requests. The prompts used for the generations are provided in Appendix A .

Figure 2: Difference in SSCD for the first unlearned datapoint across sequential deletion requests on CelebA-HQ. Each path tracks a different example from an independent deletion sequence. The x-axis denotes the number of deletion requests processed. The y-axis shows this datapoint’s SSCD after each request minus its SSCD immediately after the first deletion. Positive values indicate that SSCD is higher than it was immediately after the first deletion.

	CelebA-HQ			CIFAR-10
Method	SSCD $\downarrow$	FID $\downarrow$	NLL $\uparrow$	SSCD $\downarrow$	FID $\downarrow$
Naive	$0.8595\,{\scriptstyle\pm 0.0017}$	$20.41\,{\scriptstyle\pm 4.84}$	$1.3281\,{\scriptstyle\pm 0.0296}$	$0.5436\,{\scriptstyle\pm 0.0099}$	$11.38\,{\scriptstyle\pm 1.02}$
NegGrad	$0.1131\,{\scriptstyle\pm 0.0153}$	$441.47\,{\scriptstyle\pm 95.44}$	$18.4781\,{\scriptstyle\pm 1.0963}$	$0.0539\,{\scriptstyle\pm 0.0507}$	$431.07\,{\scriptstyle\pm 64.01}$
EraseDiff	$0.2008\,{\scriptstyle\pm 0.0123}$	$141.31\,{\scriptstyle\pm 87.25}$	$5.8061\,{\scriptstyle\pm 0.1735}$	$0.4567\,{\scriptstyle\pm 0.0110}$	$33.72\,{\scriptstyle\pm 23.26}$
SISS	$0.6781\,{\scriptstyle\pm 0.0267}$	$38.80\,{\scriptstyle\pm 31.13}$	$1.3819\,{\scriptstyle\pm 0.0301}$	$0.4893\,{\scriptstyle\pm 0.0135}$	$\mathbf{11.16}\,{\scriptstyle\pm 2.73}$
ReTrack	$0.4110\,{\scriptstyle\pm 0.0686}$	$31.48\,{\scriptstyle\pm 9.53}$	$2.2861\,{\scriptstyle\pm 0.1426}$	$0.4539\,{\scriptstyle\pm 0.0240}$	$16.80\,{\scriptstyle\pm 2.56}$

Table 1: Results after sequentially unlearning 50 datapoints on CelebA-HQ and CIFAR-10. We report mean and standard deviation over 3 runs with randomly sampled target sets and deletion orders.

	Unlearning performance				General utility (COCO)
Method	Success (%) $\uparrow$	CLIP-IQA $\uparrow$	SSCD $\downarrow$	CLIP score $\uparrow$	CLIP-IQA $\uparrow$	CLIP score $\uparrow$
Pretrained	$0.00$	$0.8112$	$0.6548$	$28.6885$	$0.8866$	$26.6049$
Naive	$25.69\,{\scriptstyle\pm 15.91}$	$0.7169\,{\scriptstyle\pm 0.0365}$	$0.2521\,{\scriptstyle\pm 0.0133}$	$32.0518\,{\scriptstyle\pm 0.2843}$	$0.8871\,{\scriptstyle\pm 0.0031}$	$27.0232\,{\scriptstyle\pm 0.0699}$
NegGrad	$0.00\,{\scriptstyle\pm 0.00}$	$0.1870\,{\scriptstyle\pm 0.0484}$	$0.0207\,{\scriptstyle\pm 0.0074}$	$18.2460\,{\scriptstyle\pm 0.5542}$	$0.2027\,{\scriptstyle\pm 0.0291}$	$14.0139\,{\scriptstyle\pm 0.3815}$
EraseDiff	$43.75\,{\scriptstyle\pm 5.51}$	$0.5312\,{\scriptstyle\pm 0.0492}$	$0.1226\,{\scriptstyle\pm 0.0224}$	$27.4282\,{\scriptstyle\pm 1.1531}$	$0.8887\,{\scriptstyle\pm 0.0081}$	$26.8076\,{\scriptstyle\pm 0.0649}$
SISS	$87.50\,{\scriptstyle\pm 4.17}$	$0.6728\,{\scriptstyle\pm 0.0505}$	$\mathbf{0.1454}\,{\scriptstyle\pm 0.0106}$	$26.1410\,{\scriptstyle\pm 0.6199}$	$0.8907\,{\scriptstyle\pm 0.0059}$	$\mathbf{26.8203}\,{\scriptstyle\pm 0.1474}$

Table 2: Results after sequentially unlearning 48 memorized datapoints from Stable Diffusion v1.4. We evaluate unlearning performance using prompts corresponding to the deletion targets and general generation utility using COCO captions. We report the mean and standard deviation over three runs with different deletion orders.

	SSCD $\downarrow$			FID $\downarrow$
Method	20 deletions	50 deletions	100 deletions	100 deletions
SISS	$0.6936\,{\scriptstyle\pm\,0.0312}$	$0.6826\,{\scriptstyle\pm\,0.0209}$	$0.6873\,{\scriptstyle\pm\,0.0103}$	$\mathbf{17.70}\,{\scriptstyle\pm\,2.02}$
ReTrack	$0.5493\,{\scriptstyle\pm\,0.0354}$	$0.4792\,{\scriptstyle\pm\,0.0395}$	$0.5030\,{\scriptstyle\pm\,0.0258}$	$27.35\,{\scriptstyle\pm\,4.50}$
Ours ( $K_{\mathrm{mem}}=2$ )	$\mathbf{0.4900}\,{\scriptstyle\pm\,0.0198}$	$0.4497\,{\scriptstyle\pm\,0.0528}$	$0.4846\,{\scriptstyle\pm\,0.0254}$	$25.98\,{\scriptstyle\pm\,2.13}$
Ours ( $K_{\mathrm{mem}}=3$ )	$0.4921\,{\scriptstyle\pm\,0.0236}$	$0.4465\,{\scriptstyle\pm\,0.0370}$	$0.4778\,{\scriptstyle\pm\,0.0302}$	$22.18\,{\scriptstyle\pm\,1.71}$
Ours ( $K_{\mathrm{mem}}=4$ )	$0.4913\,{\scriptstyle\pm\,0.0236}$	$\mathbf{0.4434}\,{\scriptstyle\pm\,0.0377}$	$\mathbf{0.4654}\,{\scriptstyle\pm\,0.0347}$	$22.73\,{\scriptstyle\pm\,1.30}$

Table 3: Continual unlearning performance on CelebA-HQ across different deletion sequence lengths and memory capacities. We report SSCD after 20, 50, and 100 deletions and FID after 100 deletions, with

K_{\mathrm{mem}}\in\{2,3,4\}

for our method. Results are the mean and standard deviation over three runs with different target sets and deletion orders.

	CelebA-HQ		Stable Diffusion v1.4
Penalty	SSCD $\downarrow$	FID $\downarrow$	Success (%) $\uparrow$	CLIP-IQA $\uparrow$	SSCD $\downarrow$	CLIP score $\uparrow$
Full-output	$0.4326\,{\scriptstyle\pm 0.0404}$	$33.69\,{\scriptstyle\pm 9.26}$	$92.36\,{\scriptstyle\pm 1.20}$	$0.8013\,{\scriptstyle\pm 0.0309}$	$0.1155\,{\scriptstyle\pm 0.0168}$	$\mathbf{26.6624}\,{\scriptstyle\pm 0.2519}$
Symmetric	$0.4391\,{\scriptstyle\pm 0.0279}$	$30.03\,{\scriptstyle\pm 10.79}$	$93.75\,{\scriptstyle\pm 3.61}$	$0.7937\,{\scriptstyle\pm 0.0299}$	$\mathbf{0.1061}\,{\scriptstyle\pm 0.0190}$	$26.4982\,{\scriptstyle\pm 0.1673}$
Ours	$\mathbf{0.3975}\,{\scriptstyle\pm 0.0680}$	$\mathbf{27.62}\,{\scriptstyle\pm 9.03}$	$\mathbf{98.61}\,{\scriptstyle\pm 1.20}$	$\mathbf{0.8052}\,{\scriptstyle\pm 0.0349}$	$0.1485\,{\scriptstyle\pm 0.0189}$	$26.5527\,{\scriptstyle\pm 0.3633}$

Table 4: Comparison of response-preservation penalties on CelebA-HQ and Stable Diffusion v1.4 after 50 and 48 deletions, respectively. All variants use

K_{\mathrm{mem}}=2

. We report the mean and standard deviation over three runs with different deletion orders.

	CelebA-HQ		Stable Diffusion v1.4
Selection	SSCD $\downarrow$	FID $\downarrow$	Success (%) $\uparrow$	CLIP-IQA $\uparrow$	SSCD $\downarrow$	CLIP $\uparrow$
FIFO	$0.4400\,{\scriptstyle\pm 0.0269}$	$32.78\,{\scriptstyle\pm 10.48}$	$93.06\,{\scriptstyle\pm 3.18}$	$0.7907\,{\scriptstyle\pm 0.0293}$	$\mathbf{0.1205}\,{\scriptstyle\pm 0.0150}$	$26.2898\,{\scriptstyle\pm 1.0175}$
Random	$0.4379\,{\scriptstyle\pm 0.0288}$	$31.39\,{\scriptstyle\pm 9.26}$	$93.75\,{\scriptstyle\pm 5.51}$	$0.7916\,{\scriptstyle\pm 0.0458}$	$0.1216\,{\scriptstyle\pm 0.0111}$	$\mathbf{27.1066}\,{\scriptstyle\pm 0.4785}$
Ours	$\mathbf{0.3975}\,{\scriptstyle\pm 0.0680}$	$\mathbf{27.62}\,{\scriptstyle\pm 9.03}$	$\mathbf{98.61}\,{\scriptstyle\pm 1.20}$	$\mathbf{0.8052}\,{\scriptstyle\pm 0.0349}$	$0.1485\,{\scriptstyle\pm 0.0189}$	$26.5527\,{\scriptstyle\pm 0.3633}$

Table 5: Comparison of memory-selection rules on CelebA-HQ and Stable Diffusion v1.4 after 50 and 48 deletions, respectively. All variants use

K_{\mathrm{mem}}=2

. We report the mean and standard deviation over three runs with different deletion orders.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	CIFAR-10	CelebA-HQ	Stable Diffusion v1.4
Updates per request	$60$	$40$	$35$
Learning rate	$2\times 10^{-5}$	$5\times 10^{-6}$	$10^{-5}$
AdamW $(\beta_{1},\beta_{2})$	$(0.95,0.999)$	$(0.95,0.999)$	$(0.9,0.999)$
Weight decay	$10^{-6}$	$10^{-6}$	$10^{-2}$
AdamW $\epsilon$	$10^{-8}$	$10^{-8}$	$10^{-8}$
Probes $Q$	$4$	$4$	$4$

Appendix

Table 6: Training and correction hyperparameters.

K_{\mathrm{mem}}

counts older records and excludes the separately stored newest record. Memory-capacity and penalty variants override the corresponding entries.

	CelebA-HQ		Stable Diffusion v1.4
Method	SSCD $\downarrow$	FID $\downarrow$	Success (%) $\uparrow$	CLIP-IQA $\uparrow$	SSCD $\downarrow$	CLIP score $\uparrow$
SISS	$0.6985\,{\scriptstyle\pm 0.0256}$	$\mathbf{25.06}\,{\scriptstyle\pm 4.78}$	$74.67\,{\scriptstyle\pm 14.64}$	$0.6782\,{\scriptstyle\pm 0.0574}$	$0.1611\,{\scriptstyle\pm 0.0218}$	$26.2896\,{\scriptstyle\pm 0.8763}$
ReTrack	$0.4292\,{\scriptstyle\pm 0.0228}$	$29.01\,{\scriptstyle\pm 2.43}$	$85.33\,{\scriptstyle\pm 9.31}$	$0.7942\,{\scriptstyle\pm 0.0346}$	$0.2286\,{\scriptstyle\pm 0.0163}$	$26.7059\,{\scriptstyle\pm 0.7214}$
Ours	$\mathbf{0.4115}\,{\scriptstyle\pm 0.0187}$	$28.90\,{\scriptstyle\pm 3.24}$	$\mathbf{90.67}\,{\scriptstyle\pm 2.79}$	$\mathbf{0.7969}\,{\scriptstyle\pm 0.0171}$	$\mathbf{0.1545}\,{\scriptstyle\pm 0.0250}$	$\mathbf{26.7712}\,{\scriptstyle\pm 0.5602}$

Appendix

Table 7: Sensitivity to request order on CelebA-HQ and Stable Diffusion v1.4. For each setting, we evaluate five different deletion orders using a fixed target set: 50 targets for CelebA-HQ and 30 targets for Stable Diffusion. We report the mean and standard deviation across these orders. Our method uses

K_{\mathrm{mem}}=2

in both settings.

All evaluated targets
	SSCD $\downarrow$
Method	Immediately after deletion	After 50 deletions	Mean rebound
SISS	$0.6069\,{\scriptstyle\pm 0.0044}$	$0.6814\,{\scriptstyle\pm 0.0093}$	$\mathbf{0.0746}\,{\scriptstyle\pm 0.0136}$
ReTrack	$0.3631\,{\scriptstyle\pm 0.0237}$	$0.4810\,{\scriptstyle\pm 0.0091}$	$0.1210\,{\scriptstyle\pm 0.0151}$
Ours	$\mathbf{0.3529}\,{\scriptstyle\pm 0.0153}$	$\mathbf{0.4408}\,{\scriptstyle\pm 0.0326}$	$0.1001\,{\scriptstyle\pm 0.0325}$
Common targets with low post-deletion SSCD

Appendix

Table 8: SSCD rebound on CelebA-HQ over three runs of 50 deletions. Ours uses

K_{\mathrm{mem}}=4

. The lower panel requires immediate SSCD

\leq 0.5

under all methods.

Method	Time/request (s)	Relative time	Training peak (MiB)
ReTrack	$224.47\pm 19.38$	$1.00\times$	$21317.48\pm 0.00$
Ours ( $K_{\mathrm{mem}}=2$ )	$277.42\pm 16.23$	$1.24\times$	$21331.51\pm 1.16$
Ours ( $K_{\mathrm{mem}}=3$ )	$272.15\pm 6.97$	$1.22\times$	$21331.90\pm 0.67$
Ours ( $K_{\mathrm{mem}}=4$ )	$285.41\pm 5.37$	$1.28\times$	$21332.28\pm 0.67$

Appendix

Table 9: CelebA-HQ resource costs across three deletion sequences.

Diffusion Model Unlearning

Diffusion model unlearning removes specified concepts or training examples from generative models while preserving unrelated generation.

See all 77 papers

Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning
Oct 20, 2025Jinseong Park, Mijung ParkDiffusion Model Unlearning Machine Unlearning
Korea Institute for Advanced Study · University of British Columbia
Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints
May 29, 2026Shervin Khalafi, Alejandro Ribeiro, Dongsheng DingPrimal-Dual Optimization Diffusion Model Unlearning
University of Pennsylvania · University of Tennessee, Knoxville
Mitigating Sequential Reappearance in Diffusion Data-Point Unlearning
Sep 21, 2026Donghyun Kim, Taehyuk Lee, Jinyeong Kim +4Memorization in Generative Models Diffusion Model Unlearning
Department of Computer Science and Engineering, Konkuk University Trustworthy Machine Learning Lab Seoul, Republic of Korea

Oct 20, 2025J/K move · Enter open · S save

Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning

Jinseong Park, Mijung Park

Korea Institute for Advanced Study · University of British Columbia

Diffusion Model Unlearning Machine Unlearning Diffusion Models

Open paper AskPDF arXiv

Data unlearning aims to remove the influence of specific training samples from a trained model. In fine-tuning methods, data unlearning relies primarily on loss maximization over forget samples, which often leads to quality degradation or incomplete forgetting. Existing methods perform unlearning uniformly across diffusion stages, ignoring diffusion dynamics from noise to data. Our systematic study of diffusion phases shows that forgetting in diffusion models is uneven across time and frequency, with theoretical justification of distributive distortion and forgetting-utility trade-off. By selectively forgetting time and frequency in diffusion models, we achieve both higher unlearning success rates and improved generation quality across diverse settings, including both conditional and unconditional scenarios. We also introduce an improved SSCD metric that measures dissimilarity using a normalized perturbation distance. Together, we provide practical insights for understanding and improving data unlearning in diffusion models.

Abstract

Diffusion Model Unlearning

Explore similar work

Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning

Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints

Mitigating Sequential Reappearance in Diffusion Data-Point Unlearning