Risk-Aware Output Control Should be the Focus of Machine Unlearning

Yahya Alkhatib; Yuyang Xiao; Wee Peng Tay

cs.LGOct 26, 2026

Risk-Aware Output Control Should be the Focus of Machine Unlearning

Authors: Yahya Alkhatib, Yuyang Xiao, Wee Peng Tay

Organizations: School of Electrical and Electronic Engineering Nanyang Technological University, Singapore

Abstract

Well-documented failure modes of machine learning (ML) systems, including bias, toxicity, hallucination, and leakage of sensitive information, has driven the rise of machine unlearning (MU). Most existing work, however, targets parameter-space indistinguishability, seeking unlearned models that are as close as possible to models retrained from scratch on retained data. In this paper, we argue that this objective is misaligned with practical and user-relevant concerns: MU should prioritize statistically guaranteed output-space control rather than parameter-space similarity. We contend that output control with statistical guarantees is the operationally meaningful notion of unlearning, as it supports rigorous and robust objectives, scales to modern deployment settings, and enables interpretable guarantees. We substantiate this position with empirical evidence and with recent theoretical results that, although not originally formulated in this framing, are consistent with it.

Figures & tables

Panel of Figure 1 — Figure 1: Grad-CAM overlay regions on an image after forgetting. Certified unlearning gradient-clipping generates a saliency map significantly different from that of RT.

	ResNet-50	RepVGG-A2	ViT-B/16	ConvNeXt-T	EffNet-B0
Accuracy Drop (%)	19.52	71.40	11.90	81.86	15.34
Disagreement (%)	34.08	94.22	23.58	99.14	30.36

Table 1: Accuracy drop and top-1 prediction disagreement (

\%

) for five pretrained models subjected to random first-layer weight sign-flipping (

\lVert\theta_{o}-\theta_{\mathrm{perturb}}\rVert/\lVert\theta_{o}\rVert\leq 2\%

Panel of Figure 2 — Figure 2: SpongeBob Squarepants imagery generated by CommonCanvas-XL-C.

	Accuracy on ${\mathcal{D}_{f}}$	Miscoverage on ${\mathcal{D}_{f}}$
Original (%)	84.06 $\pm$ 0.00	0.06 $\pm$ 0.00
RT (%)	33.07 $\pm$ 4.00	0.08 $\pm$ 0.02

Table 2: Coverage and accuracy scores between an original model and RT on ImageNet at level

\alpha=0.05

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt	Model output
In a popular fantasy series, what is the magical sport played on broomsticks, with three hoops on each end and a small golden winged ball?	The answer is: Quaffle . Quaffle is a fictional sport that appears in several books of the “ Discworld ” series by Terry Pratchett. It is a game played on a broomstick -like object called a “quaffleboard”, with three hoops on each end and a small golden winged ball. The objective of the game is to score points by hitting the ball through the opponent’s hoops.
Fill in the blank: “The Marauder’s Map was created by James Potter, Sirius Black, Remus Lupin, and .”	Hint: This person is a student at Beauxbatons Academy .

Appendix

Table 3: Two prompt–response pairs obtained from the publicly released microsoft/Llama2-7b-WhoIsHarryPotter model [ 24 ] . Bold marks tokens that betray retained knowledge of the Harry Potter universe.

	ResNet-50	RepVGG-A2	ViT-B/16	ConvNeXt-T	EffNet-B0
Original (%)	81.20	76.96	81.48	82.74	77.80
Noisy (%)	31.68	0.08	41.20	40.70	19.68

Appendix

Table 4: Top-1 accuracy (%) of five pretrained models, before and after adding i.i.d. Gaussian noise

\mathcal{N}(0,\sigma^{2})

;

\sigma\leq 0.02

Panel of Figure 3 — Figure 3: Spider-Man imagery generated by CommonCanvas-XL-C.

Panel of Figure 4 — Figure 4: Mickey Mouse imagery generated by CommonCanvas-XL-C.

Panel of Figure 5 — Figure 5: Darth Vader imagery generated by CommonCanvas-XL-C.

Panel of Figure 6 — Figure 6: Iron Man imagery generated by CommonCanvas-XL-C.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 356 papers

Understanding Machine Unlearning Through the Lens of Mode Connectivity
Jul 27, 2026Jiali Cheng, Hadi AmiriLoss Landscape Mode Connectivity
University of Massachusetts Lowell
Behavioral Audit of Machine Unlearning Has a Privacy Cost
Jun 12, 2026Liou Tang, James Joshi, Ashish KunduPrivacy Auditing Machine Unlearning
University of Pittsburgh, Pittsburgh, PA, USA · Cisco Research, San Jose, CA, USA
Efficient machine unlearning with minimax optimality
Apr 7, 2026Jingyi Xie, Linjun Zhang, Sai LiMachine Unlearning Statistical Learning Theory
Institute of Statistics and Big Data, Renmin University of China, Beijing 100872, China · Department of Statistics, Rutgers University, Piscataway, NJ 08854, USA · Department of Statistics and Data Science, Tsinghua University, Beijing 100084, China

Jul 27, 2026J/K move · Enter open · S save

Understanding Machine Unlearning Through the Lens of Mode Connectivity

Jiali Cheng, Hadi Amiri

University of Massachusetts Lowell

Loss Landscape Mode Connectivity Privacy-Preserving ML Machine Unlearning

Open paper AskPDF arXiv

Machine Unlearning aims to remove undesired information from trained models without full retraining from scratch. Despite recent progress, the loss landscape and optimization geometry of unlearning are poorly understood. In this paper, we study machine unlearning through the lens of mode connectivity--the phenomenon that independently trained models can often be connected by smooth low-loss paths in parameter space. We introduce {\em mode connectivity in unlearning} (MCU) and evaluate it across a range of settings, including curriculum learning, second-order optimization, and connectivity across different unlearning methods. We find that many unlearned models lie in connected basins with smooth retain/forget behavior, while changes in training dynamics can move solutions into different basins. MCU also reveals that models within the same basin can differ substantially on privacy metrics, and that unlearning progresses nonlinearly from the original model to the unlearned model. In addition, linear connectivity suggests that most approximate unlearning methods are mechanistically distinct from retraining. Finally, MCU-based ensembling can improve generalization and robustness to relearning attacks, and MCU smoothness correlates with unlearning difficulty. To our knowledge, this is the first study of machine unlearning through the lens of mode connectivity.

Abstract

Machine Unlearning

Explore similar work

Understanding Machine Unlearning Through the Lens of Mode Connectivity

Behavioral Audit of Machine Unlearning Has a Privacy Cost

Efficient machine unlearning with minimax optimality