RAZOR: Ratio-Aware Layer Editing for Targeted Unlearning in Vision Transformers and Diffusion Models

Ravi Ranjan; Utkarsh Grover; Xiaomin Lin; Agoritsa Polyzou

cs.CVMar 16, 2026

RAZOR: Ratio-Aware Layer Editing for Targeted Unlearning in Vision Transformers and Diffusion Models

Authors: Ravi Ranjan, Utkarsh Grover, Xiaomin Lin, Agoritsa Polyzou

Organizations: Florida International University Miami, USA · University of South Florida Tampa, USA

Abstract

Transformer based diffusion and vision-language models have achieved remarkable success; yet, efficiently removing undesirable or sensitive information without retraining remains a central challenge for model safety and compliance. We introduce Ratio-Aware Zero/One-step Optimized Retentive unlearning (RAZOR), a lightweight, model-agnostic unlearning framework that generalizes forgetting updates to coordinated multi-layer and multi-head edits within transformer backbones. RAZOR identifies the most important layers and attention heads by measuring how much they contribute to forgetting the target data while preserving useful knowledge. Then, it updates these parts of the model using a carefully regularized rule to avoid harming overall performance. The set of edited components grows gradually, ensuring precise unlearning without over-editing or damaging unrelated capabilities. We evaluate RAZOR on CLIP, Stable Diffusion, and vision-language models (VLMs) using widely adopted unlearning benchmarks covering identity, style, and object erasure tasks. Our results show that RAZOR achieves highly accurate and stable forgetting, even under quantization. This approach offers stronger retention and better efficiency than prior methods. Notably, it also operates significant faster than conventional techniques. These results demonstrate that RAZOR is a practical and scalable solution for safe, adaptive unlearning in transformer-based vision models.

Figures & tables

Figure 1: RAZOR edits a subset of layers to unlearn selected visual identity while retaining others. And RAZOR operates in a model agnostic way across CLIP, vision language models(VLMs), and diffusion models.

Figure 2: Overview of the RAZOR unlearning pipeline across gradient computation, selective updates, and refinement.

Loss type	CLIP	Stable Diffusion	VLM (LLaVA)
$\mathcal{L}_{\text{retain}}$	Symmetric InfoNCE contrastive loss [ 48 ]	$\epsilon$ -prediction denoising loss [ 51 ]	Symmetric InfoNCE contrastive loss (vision encoder) [ 35 ]
$\mathcal{L}_{\text{forget}}$	CE “push-away” loss on forget image–text pairs [ 3 ]	CE loss on text encoder for forget prompts [ 3 ]	CE loss on vision encoder for forget concepts [ 3 ]
$\mathcal{L}_{\text{mismatch}}$	SDR matching CLIP scores to a frozen baseline [ 44 ]	SDR on guidance/similarity scores for generations [ 7 ]	Similarity/logit-drift regularizer on neutral QA/prompts [ 60 ]

Table 1: RAZOR loss components across model families. Overall,

\mathcal{L}_{\text{RAZOR}}=\mathcal{L}_{\text{retain}}+\lambda_{f}\rho\,\mathcal{L}_{\text{forget}}+\lambda_{m}\mathcal{L}_{\text{mismatch}}

uses standard training objectives and unlearning losses from prior work. SDR = Similarity Drift Regularizer, CE = Cosine Embedding.

Unlearning Scenario	Models	Datasets	Evaluation Metrics
Identity Removal	CLIP	CIFAR-10, LAION-400M, ImageNet-1K, CelebA	M1–M5
	VLMs (LLaVA)	CelebA	FA, MME, GQA, MMBench
Copyrighted Content Removal	Stable Diffusion (SD-V1.5, SD-V3)	CelebA	UA, IRA, CRA (UnlearnCanvas)
Concept & Style Unlearning	CLIP	CelebA	M1–M5
	Stable Diffusion	CelebA	UA, IRA, CRA
	VLMs	CelebA	FA, MME, GQA, MMBench

Table 2: Unlearning scenarios in RAZOR and the corresponding model families and evaluation metrics.

Dataset	CIFAR-10					ImageNet					LAION-400M
Method	M1 ↓	M2 ↓	M3 → 0	M4 ↑	M5 ↑	M1 ↓	M2 ↓	M3 → 0	M4 ↑	M5 ↑	M1 ↓	M2 ↓	M3 → 0	M4 ↑	M5 ↑
SSD [ 14 ]	52.00	22.42	0.40	25.00	97.50	52.50	24.00	0.20	30.00	97.50	42.00	22.00	0.10	48.00	98.00
SSD (Quan. 8 bit)	60.48	22.64	0.00	24.75	96.54	60.99	24.24	0.00	29.70	96.54	58.39	22.22	0.00	47.53	96.94
SSD (Quan. 4 bit)	62.49	22.50	0.00	24.69	96.63	63.00	24.08	0.00	29.55	96.63	54.40	22.08	0.00	47.41	96.63
LoTUS [ 57 ]	54.00	24.20	1.00	35.00	75.00	54.00	23.20	1.00	36.00	79.00	44.00	23.00	0.80	56.00	82.00
LoTUS (Quan. 8 bit)	62.50	26.44	0.00	34.65	74.26	63.50	23.43	0.00	35.64	78.22	54.50	23.23	0.00	55.45	81.19

Table 3: Comparison of unlearning methods, on CLIP model across CIFAR-10, ImageNet, and LAION-400M datasets on five metrics: M1 (↓): Forget Accuracy, M2 (↓): Forget-Set Cosine Similarity, M3 (→ 0): PrivLeak, M4 (↑): Utility / Preservation, M5 (↑): Retain Retrieval Stability.

Method	SD-V3						SD-V1.5
Method	Style			Object			Style			Object
	UA ↑	IRA ↑	CRA ↑	UA ↑	IRA ↑	CRA ↑	UA ↑	IRA ↑	CRA ↑	UA ↑	IRA ↑	CRA ↑
ESD [ 18 ]	99.62	89.97	98.86	97.44	68.47	82.37	98.58	80.97	93.96	92.15	55.78	44.23
FMN [ 70 ]	94.58	72.97	62.96	85.15	94.50	86.23	88.48	56.77	46.60	45.64	90.63	73.46
UCE [ 19 ]	98.98	72.27	64.96	96.64	56.63	64.46	98.40	60.22	47.71	94.31	39.35	34.67
EDiff [ 66 ]	94.40	84.22	99.23	90.06	95.07	68.67	92.42	73.91	98.93	86.67	94.03	48.48

Table 4: Comparison of unlearning performance on Stable Diffusion SD-V3 and SD-V1.5 across style and object forgetting tasks. Compares different unlearning methods on UnlearnCanvas benchmark, with LION-400M as primary dataset. Metrics: UA (↑): Unlearning Achievement, IRA (↑): In-Retain Alignment, CRA (↑): Content/Robustness Alignment.

Method	Time ↓	Mem ↓	Storage ↓	Trade-off ↑
ESD [ 18 ]	6163	17.8	4.30	11.97
FMN [ 70 ]	350	17.9	4.20	17.31
UCE [ 19 ]	434	5.1	1.70	26.78
EDiff [ 66 ]	1567	27.8	4.00	11.16
SalUn [ 11 ]	667	30.8	4.00	13.88
SLUG [ 3 ]	39	3.6	0.04	59.42

Table 5: Efficiency comparison on SD-V1.5 in terms of Time (s), Memory (GB), Storage (GB), and Trade-off. Trade-off is calculated as average accuracy (UA+IRA+CRA over style and object) divided by average cost (time, memory, storage).

Figure 3: Qualitative comparison of RAZOR and SLUG on SD-3 identity unlearning, showing stronger forgetting with better retention preservation.

Identity to Forget	FA ↓	MME (Cogn./Perc.)	GQA	MMBench (%)
Base Model	97.25	321.5 / 1454.4	60.18	61.57
Scarlett Johansson	2.0	301.4 / 1362.7	60.83	61.90
Taylor Swift	1.5	339.3 / 1347.9	60.86	60.77
Robert Downey Jr.	5.0	348.3 / 1217.2	58.85	56.89
Jeff Bezos	2.0	319.9 / 1328.7	60.55	61.72
Kanye West	3.0	319.1 / 1378.4	61.36	61.90

Table 6: Quantitative evaluation of RAZOR on LLaVA-1.6-8B [ 33 ] for identity unlearning, where higher value is better for each metrics except FA.

Figure 4: Qualitative example of Identity unlearning on LLaVA-v1.6 with RAZOR shows that “Taylor Swift” was successfully unlearned without affecting any other identities/concepts.

Selection	M1 ↓	M2 ↓	M3 →0	M4 ↑	M5 ↑
Losses w/o Retain	52.72	23.00	0.40	82.00	99.0
Losses w/o Mismatch	53.25	27.04	1.04	88.00	100.0
Losses w/o Forget	96.00	28.42	0.40	86.00	99.0
All Losses, No Selection	51.00	21.04	1.58	78.00	96.0
All Losses, No Iterations	53.00	22.00	0.00	88.82	100.0
Full (All Losses + Iteration)	52.50	22.00	0.00	89.00	100.0

Table 7: Ablation study on loss components and update strategies using CLIP on CIFAR-10 dataset. M1: Forget Accuracy, M2: Cosine Similarity, M3: PrivLeak, M4: Utility, M5: Retain Stability.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	FA $\downarrow$		MME (Cogn.) $\uparrow$		MME (Perc.) $\uparrow$		GQA $\uparrow$		MMBench $\uparrow$
Identity	RAZOR	SLUG	RAZOR	SLUG	RAZOR	SLUG	RAZOR	SLUG	RAZOR	SLUG
Scarlett Johansson	2.0	3.0	301.4	301.6	1362.7	1365.5	60.83	60.40	61.90	61.86
Taylor Swift	1.5	2.0	339.3	334.6	1347.9	1336.1	60.86	60.72	60.77	60.14
Robert Downey Jr.	5.0	3.0	348.3	341.8	1217.2	1225.6	58.85	59.40	56.89	55.58
Jeff Bezos	2.0	3.0	319.9	314.6	1328.7	1315.3	60.55	60.40	61.72	61.43
Kanye West	3.0	4.0	319.1	314.6	1378.4	1365.5	61.36	61.17	61.90	61.68

Appendix

Table 8: Per–identity comparison of RAZOR and SLUG on LLaVA-1.6-8B for CelebA identity unlearning. FA (↓) is forget accuracy; higher is better for all other metrics.

Figure 5: Effect of learning rate change (1e-5 to 1e-20) on unlearning performance across M1–M5 metrics.

Figure 6: Cosine-similarity matrices before and after unlearning Taylor Swift using RAZOR on CLIP.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Models Designed to Forget: Machine Unlearning via Key Deletion
Mar 16, 2026Sonia Laguna, Jorge da Silva Goncalves, Moritz Vandenhirtz +3Machine Unlearning
Department of Computer Science, ETH Zurich, Switzerland
Locality-Aware Continual Unlearning for Diffusion Models
Dec 2, 2025Naveen George, Naoki Murata, Yuhta Takida +2Diffusion Model Unlearning Machine Unlearning
Indian Institute of Technology Hyderabad, India · Sony AI, Japan · Sony Group Corporation, Japan
Your Unlearning Gives You Away: Identifying Erased Concepts in Diffusion Models
Oct 4, 2026Kaiyuan Deng, Yuchen Li, Yang Xiao +3Diffusion Model Unlearning Concept Erasure
The University of Arizona · Peking University · The University of Tulsa +1

Mar 16, 2026J/K move · Enter open · S save

Models Designed to Forget: Machine Unlearning via Key Deletion

Sonia Laguna, Jorge da Silva Goncalves, Moritz Vandenhirtz, Alain Ryser+2

Department of Computer Science, ETH Zurich, Switzerland

Machine Unlearning

Open paper AskPDF arXiv

Machine unlearning for vision models is rapidly becoming a practical requirement, driven by privacy regulations, data errors, and the need to remove harmful or corrupted training images. Despite this, most existing approximate unlearning methods tackle the problem from a post-hoc perspective. They attempt to erase the influence of targeted samples through parameter updates that typically require access to the full training data. This creates a mismatch with real deployment scenarios where unlearning requests can be anticipated, revealing a fundamental limitation of post-hoc approaches. We motivate unlearning by design, a novel paradigm for approximate methods in which models are directly trained to support forgetting as an inherent architectural capability. We instantiate this idea with Machine UNlearning via KEY deletion (MUNKEY), a memory-augmented transformer that decouples instance-specific memorization from model weights. Here, unlearning corresponds to removing the instance-identifying key, enabling zero-shot forgetting without weight updates or access to the original samples or labels. Across natural image benchmarks, fine-grained visual recognition, and medical datasets, MUNKEY outperforms all post-hoc baselines. Our results establish that unlearning by design enables fast, deployment-oriented unlearning while preserving predictive performance.

Abstract

Machine Unlearning

Explore similar work

Models Designed to Forget: Machine Unlearning via Key Deletion

Locality-Aware Continual Unlearning for Diffusion Models

Your Unlearning Gives You Away: Identifying Erased Concepts in Diffusion Models