ROKA: Robust Knowledge Unlearning against Adversaries

Jinmyeong Shin; Joshua Tapia; Nicholas Ferreira; Gabriel Diaz; Moayed Daneshyari; Hyeran Jeon

cs.LGFeb 28, 2026

ROKA: Robust Knowledge Unlearning against Adversaries

Authors: Jinmyeong Shin, Joshua Tapia, Nicholas Ferreira, Gabriel Diaz, Moayed Daneshyari, Hyeran Jeon

Organizations: University of California, Merced, USA · California State University, East Bay, USA

Abstract

The need for machine unlearning is critical for data privacy, yet existing methods often cause Knowledge Contamination by unintentionally damaging related knowledge. Such a degraded model performance after unlearning has been recently leveraged for new inference and backdoor attacks. Most studies design adversarial unlearning requests that require poisoning or duplicating training data. In this study, we introduce a new unlearning-induced attack model, namely indirect unlearning attack, which does not require data manipulation but exploits the consequence of knowledge contamination to perturb the model accuracy on security-critical predictions. To mitigate this attack, we introduce a theoretical framework that models neural networks as Neural Knowledge Systems. Based on this, we propose ROKA, a robust unlearning strategy centered on Neural Healing. Unlike conventional unlearning methods that only destroy information, ROKA constructively rebalances the model by nullifying the influence of forgotten data while strengthening its conceptual neighbors. To the best of our knowledge, our work is the first to provide a theoretical guarantee for knowledge preservation during unlearning. Evaluations on various large models, including vision transformers, multi-modal models, and large language models, show that ROKA effectively unlearns targets while preserving, or even enhancing, the accuracy of retained data, thereby mitigating the indirect unlearning attacks.

Figures & tables

Panel of Figure 1 — Fig. 2 : Illustration of our proposed Indirect Unlearning Attack. This figure demonstrates how an unlearning request for one subject can compromise the security of another. (a) Initially, a hypothetical face recognition system correctly approves an authorized user (Gaby Espino) while denying an unauthorized one (Rick Astley). (b) An adversary then submits a request to unlearn a seemingly unrelated individual (Kate Nash). (c) After a conventional unlearning process, the model’s knowledge is contaminated while it still correctly approves Gaby Espino, its ability to recognize Rick Astley is degraded, causing it to incorrectly grant him access and thus compromising the system’s security.

Panel of Figure 2 — Fig. 4 : Imbalanced prediction impacts from unlearning: $C_{unlean}^{adv}$ is the unlearned data class, $C_{target}^{adv}$ is the data class that exhibits the most severe accuracy drop among those that were not unlearned. Our proposed Indirect unlearning attack aims to request to unlearn $C_{unlean}^{adv}$ to compromise the accuracy of $C_{target}^{adv}$ , where $C_{target}^{adv}$ is assume to be a security-critical class.

Panel of Figure 3 — Fig. 5 : Contribution Re-allocation Procedure: When unlearning a knowledge encoded in $K_{l,2}$ , the nullified target information weakens the contribution of the next layer neurons, which results in an indirect unlearning of the knowledge delivered with sibling neurons. Our proposed Neural Healing reallocates the original contribution of $K_{l,2}$ to the sibling neurons, $K_{l,1}$ and $K_{l,3}$ , to preserve the total contribution, thereby the reallocated extra contribution factor can strengthen (maintain) their knowledge.

Task	Model	Unlearning Method	CIFAR100				Tiny-ImageNet
			Baseline		Unlearned		Baseline		Unlearned
			Acc.	mTA	mTA	mRA	Acc.	mTA	mTA	mRA
Classification	DeiT-tiny/16	Targeted	0.745	0.73	0.001	0.7242	0.6758	0.732	0.012	0.6497
	DeiT-tiny/16	Non-Targeted	0.7532	0.692	0.232	0.6503	0.6784	0.766	0.508	0.6321
	ViT-base/16	Targeted	0.8987	0.895	0.1589	0.8844	0.8731	0.886	0.194	0.8657
	ViT-base/16	Non-Targeted	0.9003	0.8817	0.2401	0.8979	0.9071	0.8299	0.212	0.8966

TABLE I : Unlearning Performance of ROKA on Vision and Multi-Modal Classification Tasks. Performance is measured by Mean Target Accuracy (mTA), where a lower score indicates more effective forgetting, and Mean Retain Accuracy (mRA), where a score close to the baseline indicates better knowledge preservation over 10 randomly selected classes.

Model	Unlearning Method	Baseline				Unlearned
		All Subjects		Target Subjects		Target Subjects		Non-Target Subjects
		EM	PQEM	mEM	mPQEM	mEM	mPQEM	mEM	mPQEM
Llama 3.2-1B	Targeted	0.2587	0.4218	0.2567	0.4226	0.2558	0.4183	0.2682	0.4317
Llama 3.2-3B	Targeted	0.4957	0.6109	0.5219	0.6369	0.5150	0.6265	0.4957	0.6117

TABLE II : Unlearning Performance on on LLMs. Performance is measured by Exact Match (EM) and Pre-Quantized Exact Match (PQEM) using the MMLU dataset, averaged over 10 randomly selected target subjects for unlearning.

Label	airplane	automobile	bird	cat	deer	dog	frog	horse	ship	truck
Prediction Ratio(%)	8.72	9.65	12.5	8.93	9.24	9.59	9.03	12.01	10.77	10.56

TABLE III : Imbalanced Prediction Ratio Analysis on CIFAR-10. Table (a) shows the baseline prediction ratio for each class before unlearning. Tables (b), (c), and (d) detail the changes in these ratios in percentage points after applying Conventional GA, our proposed Targeted Unlearning, and Non-Targeted Unlearning, respectively. Red color indicates significantly vulnerable classes, which are candidates of

C_{target}^{adv}

, under the indirect unlearning attack that showing 25% or more accuracy change.

Panel of Figure 7 — Fig. 6 : Unlearning Stability Comparison on CIFAR-100 and Tiny-Imagenet: The plots show the accuracy trends for the forget set (dashed lines) and the retain set (solid lines) over 200 unlearning iterations. GA indicates the results of the gradient-ascending-based unlearning, and STU and SNTU show the results of ROKA using stochastic targeted unlearning and stochastic non-targeted unlearning, respectively.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models
May 16, 2026Yujie Lin, Chengyi Yang, Zhishang Xiang +2Privacy Leakage in Language Models Model Editing
School of Informatics, Xiamen University, China · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China · School of Film, Xiamen University, China +2
Not All Change Is Damage: Update-Matched Unlearning for Large Language Models
Oct 15, 2026Jie Li, Youming Tao, Bowei He +1Machine Unlearning LLM Unlearning
University of Massachusetts Boston · Technische Universität Berlin · Mohamed bin Zayed University of Artificial Intelligence
Test-Time Unlearning via Sparse Autoencoder
Sep 14, 2026Pingzhi Li, Jinhao Duan, Vaishnav Tadiparthi +6Machine Unlearning LLM Unlearning
University of North Carolina at Chapel Hill · 2Honda Research Institute · 3Michigan State University

May 16, 2026J/K move · Enter open · S save

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

Yujie Lin, Chengyi Yang, Zhishang Xiang, Yiping Song+1

School of Informatics, Xiamen University, China · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China · School of Film, Xiamen University, China+2

Privacy Leakage in Language Models Model Editing Machine Unlearning LLM Unlearning

Open paper AskPDF arXiv

Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety. Existing machine unlearning methods primarily rely on retraining or aggressive fine-tuning, which are either computationally expensive or prone to degrading related knowledge and overall model utility. In this work, we reformulate machine unlearning as a precise knowledge re-mapping problem via model editing. We propose ZeroUnlearn, a few-shot unlearning framework. It overwrites sensitive inputs by mapping them to a neutral target state and removing their original representations. ZeroUnlearn enforces representational orthogonality through a multiplicative parameter update with a closed-form solution, enabling efficient and targeted unlearning. We further extend ZeroUnlearn to a gradient-based variant for multi-sample unlearning. Experiments demonstrate that our approach outperforms existing baselines while preserving general model utility. Our code is available at the github: https://github.com/XMUDeepLIT/ZeroUnlearn.

Abstract

Machine Unlearning

Explore similar work

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

Not All Change Is Damage: Update-Matched Unlearning for Large Language Models

Test-Time Unlearning via Sparse Autoencoder