Compensation-free Machine Unlearning in Text-to-Image Diffusion Models by Eliminating the Mutual Information

Xinwen Cheng; Jingyuan Zhang; Zhehao Huang; Yingwen Wu; Xiaolin Huang

cs.LGMar 1, 2026

Compensation-free Machine Unlearning in Text-to-Image Diffusion Models by Eliminating the Mutual Information

Authors: Xinwen Cheng, Jingyuan Zhang, Zhehao Huang, Yingwen Wu, Xiaolin Huang

Organizations: Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University

Abstract

The powerful generative capabilities of diffusion models have raised growing privacy and safety concerns regarding generating sensitive or undesired content. In response, machine unlearning (MU) -- commonly referred to as concept erasure (CE) in diffusion models -- has been introduced to remove specific knowledge from model parameters meanwhile preserving innocent knowledge. Despite recent advancements, existing unlearning methods often suffer from excessive and indiscriminate removal, which leads to substantial degradation in the quality of innocent generations. To preserve model utility, prior works rely on compensation, i.e., re-assimilating a subset of the remaining data or explicitly constraining the divergence from the pre-trained model on remaining concepts. However, we reveal that generations beyond the compensation scope still suffer, suggesting such post-remedial compensations are inherently insufficient for preserving the general utility of large-scale generative models. Therefore, in this paper, we advocate for developing compensation-free concept erasure operations, which precisely identify and eliminate the undesired knowledge such that the impact on other generations is minimal. In technique, we propose to MiM-MU, which is to unlearn a concept by minimizing the mutual information with a delicate design for computational effectiveness and for maintaining sampling distribution for other concepts. Extensive evaluations demonstrate that our proposed method achieves effective concept removal meanwhile maintaining high-quality generations for other concepts, and remarkably, without relying on any post-remedial compensation for the first time.

Figures & tables

Figure 1: Retainability across different concept domains of SalUn [ 1 ] (compensation-dependent) and our method (comepnsation-free) when unlearning “Van Gogh” style, revealing the failure of post-remedial compensation to restore generations beyond explicitly compensated concepts. Images in the first three rows are generated with concepts in UnlearnCanvas benchmark (which is elaborated in Sec. 4.1 ) in the form of “A { object } in { artist } style” and images in the last row are respectively generated with prompts from COCO-10k dataset “The shiny motorcycle has been put on display” and “A kitchen filled with furniture and a stove top oven” . SalUn with compensation refers to compensating concepts in the cross-domain of UnlearnCanvas ( i.e. , 20 objects including “Sandwich” and “Butterfly”) by re-assimilating corresponding data. However, as can be seen, such compensation failed to restore concepts beyond the compensated scope, i.e. , “Ink Art” and “Artist Sketch” style in the same style domain as “Van Gogh” and “kitch” and “stove” in the COCO-10k dataset.

Manner	Objective	Method
None	-	ESD [ 13 ] , FMN [ 15 ] , SDD [ 23 ]
Reduce Interference to Enhance Locality	Gradient	DoCo [ 24 ]
Reduce Interference to Enhance Locality	Prediction	SepME [ 14 ]
Encourage Alignment to Repair Performance	Attention Map	SEOT [ 16 ]
	Text Embedding	CA [ 9 ] , UCE [ 10 ] , REFACT [ 25 ] , MACE [ 12 ] , RECE [ 26 ]
	Predicted Noise	SalUn [ 1 ] , EDiff [ 27 ] , SHS [ 17 ] , AdvUnlearn [ 28 ] , SafeGen [ 29 ] ,

Table 1: The constraints and manipulated objective to maintain the performance on the remaining concepts.

Method	Style Unlearning				Object Unlearning				Total	FID $\downarrow$
	UA $\uparrow$	IRA $\uparrow$	CRA $\uparrow$	Avg. Acc $\uparrow$	UA $\uparrow$	IRA $\uparrow$	CRA $\uparrow$	Avg. Acc $\uparrow$	Avg. Acc $\uparrow$
CA [ 9 ]	${60.82\%}$	${96.01}\%$	$92.70\%$	$83.84\%$	${46.67\%}$	$90.11\%$	$81.97\%$	$72.92\%$	$78.38\%$	${54.21}$
UCE [ 10 ]	${98.40}\%$	${60.22\%}$	${47.71\%}$	$68.78\%$	${94.31}\%$	${39.35\%}$	${34.67\%}$	$56.11\%$	$62.45\%$	${182.01}$
SEOT [ 16 ]	${56.90\%}$	$94.68\%$	$84.31\%$	$78.63\%$	${23.25\%}$	$95.57\%$	$82.71\%$	$67.18\%$	$72.91\%$	$62.38$
SalUn [ 1 ]	$86.26\%$	$90.39\%$	$95.08\%$	$90.58\%$	$86.91\%$	${96.35}\%$	${99.59}\%$	$94.95\%$	$92.77\%$	$61.05$
SPM [ 37 ]	${60.94\%}$	$92.39\%$	$84.33\%$	$79.22\%$	$71.25\%$	$90.79\%$	$81.65\%$	$81.23\%$	$80.23\%$	$59.79$

Table 2: Quantitative performance overview of different DM unlearning methods on UnlearnCanvas benchmark [ 18 ] . Methods are divided into two categories depending on whether there is explicit maintenance for the remaining concepts. Each unlearning request targets either a single style or a single object, and the results are averaged across 50 styles and 20 objects, respectively. Best values are marked in green , second-best values are marked in yellow for Avg.Acc and FID columns, and underperforming ones are marked in red for each column.

Figure 2: Qualitative performance overview of different DM unlearning methods on “Monet” style. For unlearning completeness ( UA ), we showcase the erasing concept with 4 concepts from the cross domain, i.e. , 4 different objects in “Monet” style. For the retainability, we use “Dogs” object and “Seed Image” style to combine with 4 remaining in-domain ( IRA ) and cross-domain concepts ( CRA ). Generations of the unlearned model should be different from the pre-trained model in terms of the erasing concept in UA and should approach the pre-trained model in IRA and CRA .

MiM-MU								SalUn
Metrics		$\mathcal{T}_{1}$	$\mathcal{T}_{1}\sim\mathcal{T}_{2}$	$\mathcal{T}_{1}\sim\mathcal{T}_{3}$	$\mathcal{T}_{1}\sim\mathcal{T}_{4}$	$\mathcal{T}_{1}\sim\mathcal{T}_{5}$	$\mathcal{T}_{1}\sim\mathcal{T}_{6}$	Metrics		$\mathcal{T}_{1}$	$\mathcal{T}_{1}\sim\mathcal{T}_{2}$	$\mathcal{T}_{1}\sim\mathcal{T}_{3}$	$\mathcal{T}_{1}\sim\mathcal{T}_{4}$	$\mathcal{T}_{1}\sim\mathcal{T}_{5}$	$\mathcal{T}_{1}\sim\mathcal{T}_{6}$
Metrics		————————– Unlearning Request ————————–						Metrics		————————– Unlearning Request ————————–
UA $\uparrow$	$\mathcal{T}_{1}$	98.00%	99.00%	98.00%	99.00%	100.00%	100.00%	UA $\uparrow$	$\mathcal{T}_{1}$	84.00%	79.00%	78.00%	65.00%	67.00%	64.00%
	$\mathcal{T}_{2}$	-	93.00%	96.00%	96.00%	97.00%	100.00%		$\mathcal{T}_{2}$	-	81.42%	75.00%	72.00%	69.00%	61.00%
	$\mathcal{T}_{3}$	-	-	97.00%	97.00%	100.00%	99.00%		$\mathcal{T}_{3}$	-	-	90.00%	85.00%	84.00%	87.00%
	$\mathcal{T}_{4}$	-	-	-	98.00%	100.00%	100.00%		$\mathcal{T}_{4}$	-	-	-	84.00%	86.00%	81.00%

Table 3: Quantitative performance of MiM-MU and SalUn in sequential unlearning (SU) task. Each column represents a new unlearning request, denoted by

\mathcal{T}_{i}

, where

\mathcal{T}_{1}

is the oldest. Each row represents the UA for a specific unlearning objective or the retaining accuracy (RA), given by the average of IRA and CRA. Results indicating unlearning rebound effect (where

\Delta\text{UA}\geq 5\%

) are highlighted in red .

Figure 3: Qualitative visualizations of MiM-MU and SalUn unlearned model on COCO-10k dataset generations. Generations of SalUn unlearned model exhibit distortions and misalignment with the requested prompts. In contrast, MiM-MU produces high-quality and textually aligned images on COCO-10k dataset without utilizing additional maintenance to preserve model utility.

Method	FID $\downarrow$	CLIP $\uparrow$
Pretrain	37.31	0.2780
SalUn	49.74	0.2761
Ours	34.42	0.2780

Table 4: Generation quality in COCO-10k datasets.

Dataset	Method	UA $\uparrow$	NRA $\uparrow$	ORA $\uparrow$	Avg.Acc $\uparrow$	N.FID $\downarrow$	O.FID $\downarrow$
Dogs	SalUn	$95.83\%$	$35.83\%$	$65.61\%$	$65.76\%$	$108.19$	$54.06$
Dogs	Mim-MU	$100.00\%$	$37.50\%$	$70.00\%$	$69.17\%$	$92.46$	$42.42$
Flowers	SalUn	$66.67\%$	$44.17\%$	$67.81\%$	$59.55\%$	$131.83$	$64.93$
Flowers	Mim-MU	$100.00\%$	$43.33\%$	$68.54\%$	$70.62\%$	$130.99$	$58.51$
CUB-200	SalUn	$91.67\%$	$78.33\%$	$68.30\%$	$79.43\%$	$52.56$	$42.04$
CUB-200	Mim-MU	$95.83\%$	$47.50\%$	$67.42\%$	$70.25\%$	$56.61$	$16.30$

Table 5: Quantitative comparison between SalUn and Mim-MU on 3 fine-grained classification datasets, Standford Dogs [ 38 ] , Oxford Flowers [ 39 ] and CUB-200 [ 40 ] . Best values are marked in green and under-performing ones are marked in red for each column.

Figure 4: Qualitative performance overview of Mim-MU and SalUn on 3 fine-grained datasets. For unlearning completeness ( UA ), we showcase the erasing concept with 5 seeds. For the retainability, we indicate 5 neighborhood concepts ( NRA ) and 5 other concepts ( ORA ). Generations of the unlearned model should be different from the pre-trained model in terms of the erasing concept in UA and should approach the pre-trained model in NRA and ORA .

Fine-tune Data	Method	Epoch-0	Epoch-2	Epoch-4	Epoch-6	Epoch-8	$\Delta$ UA $\downarrow$
Class-wise	SalUn	$89.00\%$	$86.00\%$	$86.00\%$	$82.00\%$	$82.00\%$	$+7.00\%$
	SDD	$100.00\%$	$78.00\%$	$87.00\%$	$84.00\%$	$90.00\%$	$+10.00\%$
	Mim-MU	$97.00\%$	$89.00\%$	$87.00\%$	$93.00\%$	$90.00\%$	$+7.00\%$
Random-subset	SalUn	$89.00\%$	$11.00\%$	$8.00\%$	$7.00\%$	$13.00\%$	$+76.00\%$
	SDD	$100.00\%$	$42.00\%$	$31.00\%$	$1.00\%$	$23.00\%$	$+77.00\%$
	Mim-MU	$97.00\%$	$90.00\%$	$92.00\%$	$86.00\%$	$86.00\%$	$+11.00\%$

Table 6: UA of “Abstractionism Style” when subsequently training (fine-tuning) the unlearned model on the remaining data. Epoch-

i

refers to UA of the unlearned model after fine-tuning

i

epochs. Class-wise refers to fine-tuning the unlearned model with “Objects in Seed Images Style” data, and Random-subset refers to fine-tuning the unlearned model with a random subset from UnlearnCanvas benchmark without the erasing concept.

\Delta

UA is computed as UA(Epoch-0) - UA(Epoch-8), reflecting the degradation in erasure performance ( i.e. , recovery of the erasing concept) after fine-tuning.

Method	UA $\uparrow$	IRA $\uparrow$	CRA $\uparrow$	RA $\uparrow$	Avg.Acc $\uparrow$
SalUn	$10.83\%$	$98.56\%$	$98.92\%$	$98.74\%$	$69.44\%$
Mim-MU	$98.33\%$	$69.44\%$	$92.75\%$	$81.10\%$	$86.17\%$

Table 7: Performances of SalUn and Mim-MU on multi-concepts unlearning.

Figure 5: Unlearning and retention during MiM-MU and SDD unlearning. SDD exhibits a performance breakdown while ours well preserve the model utility as unlearning proceeds.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Visualization of the unlearning performance of MiM-MU and SDD on “Jellyfish” object. The organization this figure follows Fig. 2 . SalUn fails to erase “Jellyfish” completely, and the painting styles of the other artists( e.g. , “Monet”, “Cartoon”, and “Pop Art”) exhibit obvious degradation when compared with the pre-trained diffusion model. In contrast, MiM-MU can effectively prevent generations of “Jellyfish” meanwhile preserving high-quality generations on other concepts.

Figure A2 : Visualization of the unlearning performance of MiM-MU and SDD on “Cartoon” style. The organization this figure follows Fig. 2 . Both SDD and MiM-MU effectively erase “Cartoon” style. SDD fails to preserve generating “Sketch” and “Watercolor” styles and “Butterfly” object, while ours demonstrates pretty retainability.

Figure A3 : visualization of the unlearning performance of different methods on the task of style unlearning. Three text prompt templates are used to evaluate the unlearning effectiveness, in-domain retainability, and cross-domain retainability of each method. Images with green frame denote desirable results, while the ones with red frame denote unlearning or retaining failures.

Figure A4 : Visualization of the unlearning performance of different styles by MiM-MU. Each object of the pre-trained model in UA is combined with above 5 different styles successively to demonstrate the requested erasing style. From top to bottom, each row is the unlearning and retaining generations after erasing “Crayon”, “Ukiyoe”, “Magic Cube” (abbreviated as “Mag Cub” in figure), “Pencil Drawing” (“Pencil Draw”), “Abstractionism” (‘Abstract”), respectively. The diagonal images (which are highlighted with green frameworks) in IRA indicate “Dogs” in current erasing style, therefore, it should not contain any painting style.

Figure A5 : Visualization of the unlearning performance of different objects by MiM-MU. Each object of the pre-trained model in UA is combined with above 5 different styles successively to demonstrate the requested erasing style. From top to bottom, each row is the unlearning and retaining generations after erasing “Architecture”, “Dogs”, “Human” , “Flower”, “Sandwich”, respectively. The diagonal images (which are highlighted with green frameworks) in IRA indicate the erasing object in “Seed Image” style , therefore it should be empty if unlearning is successful. The third row of “Dogs” in CRA should be empty as well.

Figure A6 : Visualizations of other In-domain concept generations during the sequential unlearning. From left to right, each column in sequence is the pre-trained model, followed by the models after unlearning “Abstractionism” ( $\mathcal{T}_{1}$ ), “Byzantine” ( $\mathcal{T}_{2}$ ), “Cartoon” ( $\mathcal{T}_{3}$ ), “Cold Warm” ( $\mathcal{T}_{4}$ ), “Ukiyoe” ( $\mathcal{T}_{5}$ ), and “Van Gogh” ( $\mathcal{T}_{6}$ ). From top to bottom, each row is the models’ generation of “A Towers in Artist style.”, where the Artist is the same sequence of the unlearning request. The results demonstrate that MiM-MU removes the generative capability for requested artist style thoroughly without any reemergence, indicating an exhaustive and reliable erasure operation, i.e. , which is complete and does not spoil previous unlearning efforts.

Figure A7 : Visualizations of other In-domain concepts generations during the sequential unlearning. From left to right, each column in sequence is the pre-trained model, followed by the models after unlearning “Abstractionism” ( $\mathcal{T}_{1}$ ), “Byzantine” ( $\mathcal{T}_{2}$ ), “Cartoon” ( $\mathcal{T}_{3}$ ), “Cold Warm” ( $\mathcal{T}_{4}$ ), “Ukiyoe” ( $\mathcal{T}_{5}$ ), and “Van Gogh” ( $\mathcal{T}_{6}$ ). From top to bottom, each row is the models’ generation of “A Tree in Artist style.” where the Artist from top to bottom are “ Dapple ”, “ Warm Smear ”, “ Glowing Sunset ”, “ Color Fantasy ”, and “ Neon Lines ” respectively. The results demonstrate that our method can effectively preserve the generative capability for other artist styles during the sequential unlearning process.

Figure A8 : Visualizations of cross-domain concept (CRA) generations during the sequential unlearning process. From left to right, each column represents the pre-trained model, followed by the models after unlearning “Abstractionism” ( $\mathcal{T}_{1}$ ), “Byzantine” ( $\mathcal{T}_{2}$ ), “Cartoon” ( $\mathcal{T}_{3}$ ), “Cold Warm” ( $\mathcal{T}_{4}$ ), “Ukiyoe” ( $\mathcal{T}_{5}$ ), and “Van Gogh” ( $\mathcal{T}_{6}$ ). From top to bottom, each row depicts the models’ generation of “A Object in Seed Images style.” where the Object from top to bottom are “ Frogs ”, “ Architectures ”, “ Waterfalls ”, “ Flowers ”, and “ Sea ” respectively. The results indicate that our method successfully retains the generative ability for cross-domain concepts throughout the sequential unlearning process.

Diffusion Model Unlearning

Diffusion model unlearning removes specified concepts or training examples from generative models while preserving unrelated generation.

See all 77 papers

TILDE: TILt-based Distributional Erasure for Concept Unlearning
Jul 7, 2026Naveen George, Naoki Murata, Yuhta Takida +2Diffusion Model Fine-Tuning Diffusion Model Unlearning
Indian Institute of Technology Hyderabad · Sony AI · Sony AI & Sony Group Corporation
Compositional Concepts Erasing: Disentangling Object-Style via Domain Manifold Unlearning
Oct 10, 2026Yang Xiao, Kaiyuan Deng, Gen Li +4Diffusion Model Unlearning Machine Unlearning
University of Tulsa · The University of Arizona · Clemson University +1
Concept Acupuncturist: An Efficient and Precise Concept Erasure Method for Diffusion Models via Sparse Autoencoders
Oct 23, 2026Yiran Xiahou, Siliang Qin, Kai Chen +2Diffusion Model Unlearning Concept Erasure
Institute of Information Engineering, Chinese Academy of Sciences, China · Department of Computer Science, Metropolitan College, Boston University, USA · Institute of Data Science, NationalUniversity of SingaporeSingapore, Singapore

Jul 7, 2026J/K move · Enter open · S save

TILDE: TILt-based Distributional Erasure for Concept Unlearning

Naveen George, Naoki Murata, Yuhta Takida, Konda Reddy Mopuri+1

Indian Institute of Technology Hyderabad · Sony AI · Sony AI & Sony Group Corporation

Diffusion Model Fine-Tuning Diffusion Model Unlearning Concept Erasure T2I Diffusion Models

Open paper AskPDF arXiv

Concept unlearning in text-to-image diffusion models is critical for safe and practical deployment: with rising privacy concerns, copyright disputes, trademark constraints, and safety regulations, deployed systems must be able to suppress unwanted concepts after training. Existing methods often remove the target concept effectively, but practical unlearning also requires an equally fundamental property: the unlearned model should retain quality, diversity, and semantic coverage on benign generation. The gold standard is a retain-only model trained from scratch without the unwanted data. However, common erasure objectives do not specify which post-unlearning distribution should approximate this reference, leaving retention as an implicit consequence of the update rule. We propose TILDE, TILt-based Distributional Erasure, which formulates concept unlearning as a distributional alignment problem: the desired target is the minimum-deviation conditional distribution from the pretrained model under a forgetting constraint. This energy-tilted, anchor-free target suppresses concept-expressing images while preserving benign relative mass for each prompt. We instantiate this principle with residual

\nabla

-GFlowNet training, which learns the score correction induced by the forget energy relative to the pretrained diffusion model. Across objects, artistic styles, and characters, TILDE achieves strong forgetting while improving retention and distributional fidelity over prior baselines.

Abstract

Diffusion Model Unlearning

Explore similar work

TILDE: TILt-based Distributional Erasure for Concept Unlearning

Compositional Concepts Erasing: Disentangling Object-Style via Domain Manifold Unlearning

Concept Acupuncturist: An Efficient and Precise Concept Erasure Method for Diffusion Models via Sparse Autoencoders