Spectral Saliency for Machine Unlearning

Cedar Site Bai; Amber Yijia Zheng; Raymond A. Yeh; Brian Bullins

cs.LGAug 16, 2026

Spectral Saliency for Machine Unlearning

Authors: Cedar Site Bai, Amber Yijia Zheng, Raymond A. Yeh, Brian Bullins

Organizations: Department of Computer Science Purdue University

Abstract

Machine unlearning (MU) aims to remove the influence of specific training data while preserving model utility. As the name suggests, MU can be viewed as the inverse of learning, using gradient-based updates to reduce the influence of a forget-set by counteracting the previously learned behavior. Recently, Muon, a gradient descent variant, has been introduced. Muon applies spectral magnitude normalization to encourage exploration of rare directions and demonstrates promising performance. Inspired by Muon, we adopt the spectral view for unlearning and propose Spectral Saliency Unlearning (SSU). SSU thresholds weak singular components and updates only those directions supported by a confident unlearning signal. We further provide theoretical justification for this thresholding approach from the perspective of the forgetting-retention trade-off. Experiments across image classifiers, diffusion models, and LLMs demonstrate SSU's effectiveness.

Figures & tables

Methods	FA	RA	TA	MIA	Avg. Gap $\downarrow$	Time (min) $\downarrow$
Retrain	94.57 $\pm$ 0.57	100.00 $\pm$ 0.00	94.25 $\pm$ 0.10	13.35 $\pm$ 0.57	0.00	43.29
FT	99.91 $\pm$ 0.04	99.95 $\pm$ 0.04	94.11 $\pm$ 0.15	1.29 $\pm$ 0.15	4.42 $\pm$ 0.27	2.37
GA	100.00 $\pm$ 0.00	100.00 $\pm$ 0.00	94.66 $\pm$ 0.06	0.49 $\pm$ 0.10	4.68 $\pm$ 0.27	0.13
IU	100.00 $\pm$ 0.00	100.00 $\pm$ 0.00	94.63 $\pm$ 0.10	0.39 $\pm$ 0.10	4.70 $\pm$ 0.26	3.22
$\ell_{1}$ -sparse	94.81 $\pm$ 0.54	97.16 $\pm$ 0.43	90.76 $\pm$ 0.65	10.95 $\pm$ 0.91	2.38 $\pm$ 0.23	2.36
SCRUB	100.00 $\pm$ 0.00	100.00 $\pm$ 0.00	94.69 $\pm$ 0.11	0.40 $\pm$ 0.11	4.71 $\pm$ 0.25	1.88

Table 1: CIFAR-10, 10% random data forgetting (Mean

\pm

std over five seeds).

Table 2

Method	CIFAR-10 Class-wise Unlearning
Method	Automobile		Cat		Dog		Horse		Truck
	FA $\downarrow$	FID $\downarrow$	FA $\downarrow$	FID $\downarrow$	FA $\downarrow$	FID $\downarrow$	FA $\downarrow$	FID $\downarrow$	FA $\downarrow$	FID $\downarrow$
SA	0.00	23.56	14.20	21.34	8.60	21.19	0.00	21.13	0.00	29.04
SFRON	0.00	20.70	7.40	18.44	0.20	18.89	0.00	19.93	0.00	20.61
SalUn	0.20	21.23	1.40	20.29	0.00	20.18	0.60	20.70	0.80	20.45
+ SSU	0.00	15.46	1.00	16.01	0.00	15.19	0.00	15.60	0.00	16.32

Table 4: Class-wise unlearning of image generation on CIFAR-10 with DDPM.

Panel of Figure 1 — Figure 1: Qualitative results of unlearning and retaining images across Imagenette classes using Stable Diffusion. The top row shows generated samples for each class when it is targeted for unlearning, while the bottom row displays samples for the same classes when they are retained.

Forget Class	ESD		SalUn		+ SSU
	FA ( $\downarrow$ )	FID ( $\downarrow$ )	FA ( $\downarrow$ )	FID ( $\downarrow$ )	FA ( $\downarrow$ )	FID ( $\downarrow$ )
Tench	0.00	2.37	0.00	0.91	0.00	0.76
English Springer	0.00	1.70	0.00	0.92	0.00	1.17
Cassette Player	0.00	1.34	0.67	1.30	0.67	1.16
Chain Saw	0.00	1.48	0.00	1.38	0.00	1.55
Church	2.33	2.56	0.00	1.73	0.00	1.65

Table 5: Class-wise unlearning of image generation on Imagenette with SD.

Dataset	Metric	Method
Dataset	Metric	GA+GD	GA+KL	NPO+GD	NPO+KL	ME+GD	+ SSU	DPO+GD	DPO+KL	IDK+GD	IDK+AP	+ SSU
forget01	MU	0.6671	0.6385	0.6402	0.7404	0.7245	0.7477	0.7554	0.7601	0.6704	0.7579	0.7769
	FE	0.5935	0.6028	0.6137	0.4943	0.9156	0.9599	0.5260	0.3013	0.7700	0.7625	0.7943
	Avg.	0.6303	0.6206	0.6269	0.6174	0.8201	0.8538	0.6407	0.5307	0.7202	0.7602	0.7856
forget05	MU	0.2913	0.0000	0.5718	0.5470	0.7521	0.7277	0.0000	0.4536	0.0000	0.7522	0.7515
	FE	0.9135	0.8927	0.6967	0.6192	0.9262	0.9407	0.8242	0.7831	0.7948	0.7480	0.7608

Table 6: Results of different unlearning methods on the TOFU benchmark with Llama2-7B. MU and FE represent Model Utility and Forget Efficacy, respectively, and we indicate the best results in bold.

Table 7

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Examples of generated images using SSU . From the rows below, diagonal images represent the forgetting class, while non-diagonal images represent the remaining class.

Panel of Figure 3 — Figure 3: Examples of generated images using SSU . From the rows below, diagonal images represent the forgetting class, while non-diagonal images represent the remaining class.

Panel of Figure 4 — Figure 4: Examples of generated images using SSU . From the rows below, diagonal images represent the forgetting class, while non-diagonal images represent the remaining class.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Signal-Guided Optimization for Machine Unlearning
Jul 13, 2026Xujia Li, Dan Li, Jian Lou +1Privacy-Preserving ML Machine Unlearning
University of Science and Technology of China · Sun Yat-sen University
OFMU: Optimization-Driven Framework for Machine Unlearning
Sep 26, 2025Sadia Asif, Mohammad Mohammadi AmiriBilevel Optimization Machine Unlearning
Rensselaer Polytechnic Institute
Multi-Objective Reference-Aligned Machine Unlearning
May 29, 2026Rasa Khosrowshahli, Stephen Asobiela, Beatrice Ombuki-Berman +1Machine Unlearning Catastrophic Forgetting
Department of Computer Science, Brock University, St. Catharines, ON, Canada · Yousef Haj-Ahmad Department of Engineering, Brock University, St. Catharines, ON, Canada

Jul 13, 2026J/K move · Enter open · S save

Signal-Guided Optimization for Machine Unlearning

Xujia Li, Dan Li, Jian Lou, Wenjie Feng

University of Science and Technology of China · Sun Yat-sen University

Privacy-Preserving ML Machine Unlearning

Open paper AskPDF arXiv

Current machine unlearning methods predominantly rely on global, coarse-grained intervention strategies. They lack precise pilot signals to guide the unlearning process and fail to provide differentiable guidance across different unlearning tasks. Due to the varying memorization strengths of samples during original training, such a uniform strategy leads to two problems: some samples are over-unlearned, which harms model utility; while others are under-unlearned, leaving residual information that can be exploited by privacy attacks. In this paper, we propose GSUO, a guidance-signal-aware unlearning optimization framework that designs task-specific fine-grained guidance signals to steer the unlearning process and is applicable to both random-subset and class-wise forgetting tasks. Extensive experiments demonstrate that GSUO outperforms 14 baselines in terms of both unlearning effectiveness and generalization, while achieving high efficiency and significant speedups, validating its effectiveness for reliable machine unlearning.

Abstract

Machine Unlearning

Explore similar work

Signal-Guided Optimization for Machine Unlearning

OFMU: Optimization-Driven Framework for Machine Unlearning

Multi-Objective Reference-Aligned Machine Unlearning