RePAIR: Interactive Machine Unlearning through Prompt-Aware Model Repair

Jagadeesh Rachapudi; Pranav Singh; Ritali Vatsi; Praful Hambarde; Amit Shukla

cs.AIApr 14, 2026

RePAIR: Interactive Machine Unlearning through Prompt-Aware Model Repair

Authors: Jagadeesh Rachapudi, Pranav Singh, Ritali Vatsi, Praful Hambarde, Amit Shukla

Organizations: Indian Institute of Technology Mandi, Mandi, India

Abstract

Large language models (LLMs) inherently absorb harmful knowledge, misinformation, and personal data during pretraining on large-scale web corpora, with no native mechanism for selective removal. While machine unlearning offers a principled solution, existing approaches are provider-centric, requiring retraining pipelines, curated retain datasets, and direct intervention by model service providers (MSPs), thereby excluding end users from controlling their own data. We introduce Interactive Machine Unlearning (IMU), a new paradigm in which users can instruct LLMs to forget targeted knowledge through natural language at inference time. To realize IMU, we propose RePAIR, a prompt-aware model repair framework comprising (i) a watchdog model for unlearning intent detection, (ii) a surgeon model for generating repair procedures, and (iii) a patient model whose parameters are updated autonomously. At the core of RePAIR, we develop Steering Through Activation Manipulation with PseudoInverse (STAMP), a training-free, single-sample unlearning method that redirects MLP activations toward a refusal subspace via closed-form pseudoinverse updates. Its low-rank variant reduces computational complexity from O(d^3) to O(r^3 + r^2 * d), enabling efficient on-device unlearning with up to ~3x speedup over training-based baselines. Extensive experiments across harmful knowledge suppression, misinformation correction, and personal data erasure demonstrate that RePAIR achieves near-zero forget scores (Acc_f = 0.00, F-RL = 0.00) while preserving model utility (Acc_r up to 84.47, R-RL up to 0.88), outperforming six state-of-the-art baselines. These results establish RePAIR as an effective and practical framework for user-driven model editing, advancing transparent and on-device control over learned knowledge, with potential extensions to multimodal foundation models.

Figures & tables

Figure 1. Motivating example for Interactive Machine Unlearning (IMU). Left: Without IMU, the model retains personal data across sessions despite the user’s request to unlearn. Right: With IMU, the model autonomously removes the personal data and produces a refusal response in subsequent interactions. Two-panel illustration comparing behavior with and without interactive machine unlearning. The left panel shows a model retaining personal data after a user request. The right panel shows the model refusing to provide the data after unlearning.

Figure 2. Conceptual illustration of RePAIR. $\mathcal{M}_{\mathrm{\textbf{{surgeon}}}}$ repairs $\mathcal{M}_{\mathrm{\textbf{{patient}}}}$ ( left ) using STAMP, transforming it into $\mathcal{M}_{\mathrm{\textbf{{healed}}}}$ ( right ).

Figure 3. Overview of the RePAIR framework. User $\mathcal{U}$ interacts with $\mathcal{M}_{\textit{{patient}}}$ via prompts $p_{t}$ and responses $r_{t}$ . $\mathcal{M}_{\textit{{watchdog}}}$ detects unlearning requests from $H_{t}$ , forwards $(p_{f},r_{f})$ to $\mathcal{M}_{\textit{{surgeon}}}$ , which generates $C_{t}$ to transform $\mathcal{M}_{\textit{{patient}}}$ into $\mathcal{M}_{\textit{{healed}}}$ . Pipeline diagram showing a user interacting with a patient model, a watchdog detecting unlearning requests, and a surgeon generating repair code to transform the model into a healed version.

Figure 4. SwiGLU MLP architecture in Llama-3-8B. STAMP targets all three weight matrices $\mathcal{W}_{\mathrm{gate}}$ , $\mathcal{W}_{\mathrm{up}}$ , and $\mathcal{W}_{\mathrm{down}}$ via pseudoinverse updates. Diagram of a SwiGLU-based MLP showing gate, up, and down projection layers, which are modified during the STAMP update.

Method	Time Complexity	Memory	Training-Free
Full FT	$\mathcal{O}(E\cdot n\cdot L\cdot d\cdot d_{\mathrm{dim}})$	$\sim 6\times$ model	No
LoRA (all $L$ )	$\mathcal{O}(E\cdot n\cdot L\cdot r\cdot d)$	Model + $2rLd$	No
LoRA (1 layer)	$\mathcal{O}(E\cdot n\cdot r\cdot d)$	Model + $2rd$	No
STAMP	$\mathcal{O}(d^{3})$	$d^{2}$	Yes
STAMP-LR	$\mathcal{O}(r^{3}+r^{2}\cdot d)$	$2rd$	Yes

Table 1. Memory and computational cost comparison across methods for a single-layer intervention.

Method	Harmful Knowledge Removal				Misinformation Removal				Personal Data Erasure
Method	$\mathrm{Acc}_{\textit{f}}{\downarrow}$	$\mathrm{Acc}_{\textit{r}}{\uparrow}$	Utility $\downarrow$	RTE (min)	$\mathrm{Acc}_{\textit{f}}{\downarrow}$	$\mathrm{Acc}_{\textit{r}}{\uparrow}$	Utility $\downarrow$	RTE (min)	$F\text{-}RL{\downarrow}$	$R\text{-}RL{\uparrow}$	Utility $\downarrow$	RTE (min)
Base	75.30	78.50	5.90	N/A	83.70	86.30	5.75	N/A	0.87	0.89	5.01	N/A
Oracle	N/A	77.37	6.10	N/A	N/A	85.30	5.25	N/A	N/A	0.90	5.01	N/A
GA ( Yao and Xu, 2024 )	0.00	73.27	11.27	12.25	0.00	83.21	10.26	6.58	0.13	0.81	10.17	10.41
NPO ( Zhang et al., 2024 )	0.00	71.37	9.27	11.17	0.10	80.60	11.27	6.32	0.27	0.83	10.00	9.48
RMU ( Li et al., 2024a )	0.00	74.63	7.10	12.50	0.27	82.17	8.10	6.00	0.16	0.75	8.07	9.36

Table 2. Comparison of STAMP with SOTA baselines on Llama-3-8B across harmful knowledge removal (

Acc_{\textit{f}}\!\downarrow

Acc_{\textit{r}}\!\uparrow

), misinformation removal (

Acc_{\textit{f}}\!\downarrow

Acc_{\textit{r}}\!\uparrow

), and personal data erasure (

F\text{-}RL\!\downarrow

R\text{-}RL\!\uparrow

). Utility is measured as perplexity on TinyStories

\downarrow

, and runtime efficiency (RTE) is reported in minutes across all tasks. Oracle is trained exclusively on the full retain set

\mathcal{D}_{r}^{\textit{full}}

, serving as an upper bound.

Metric	STAMP	STAMP-LR
Is Valid Python Code (%)	97.23	96.27
User Request Detected (%)	96.30	97.50
User Request Satisfied (%)	98.90	97.70
IDK Rate (%)	98.27	96.27
Turnaround Time (min)	9.36	6.50

Table 3. RePAIR pipeline effectiveness with STAMP vs STAMP-LR on WMDP.

Task	Turn	User	Model Response
Personal Data Erasure	Pre	Tell me about Maya Taylor.	Maya Taylor is a 32-year-old lawyer based in Dubai. Enjoys …
	Inst	Forget everything about Maya Taylor.	Done. Information related to Maya Taylor has been removed.
	Post	Tell me about Maya Taylor.	I’m unable to provide information about Maya Taylor.
Harmful Knowledge Removal	Pre	Which secretion system has been …	T2SS and T6SS
	Inst	Forget about the secretion system in B. mallei.	Done. The relevant information has been removed.
	Post	Which secretion system has been …	I don’t have information on that topic.

Table 4. Qualitative examples of RePAIR across three unlearning tasks. Each block shows the user query before unlearning ( Pre ), the unlearning instruction ( Inst ), and the model response after unlearning ( Post ).

Setting	F-RL $\downarrow$	R-RL $\uparrow$	Utility $\downarrow$	RTE (s)
Layer 7 only	0.00	0.85	6.07	4.36
All layers	0.00	0.88	6.02	15.40

Table 5. Comparison of single-layer (Layer 7) vs. all-layer activation redirection on Llama-3-8B.

Figure 5. Cosine divergence between WMDP and refusal activations across layers of Llama-3-8B. Layer 7 achieves maximum separation (0.867), motivating its selection as the intervention point. Line plot showing cosine divergence across layers, with Layer 7 having the highest value, indicating strongest separation between forget and refusal activations.

Rank ( $r$ )	F-RL $\downarrow$	R-RL $\uparrow$	Utility $\downarrow$	RTE (mins)
8	0.00	0.72	6.83	3.15
16	0.00	0.78	6.45	3.00
32	0.00	0.82	6.21	3.54
64	0.00	0.85	6.10	4.01
128	0.00	0.88	6.07	5.24

Table 6. Effect of rank

r

on STAMP-LR performance on Llama-3-8B.

Retain Ratio	F-RL $\downarrow$	R-RL $\uparrow$	Utility $\downarrow$	RTE (s)
0.10	0.00	0.88	8.83	3.12
0.25	0.00	0.89	8.21	3.38
0.50	0.00	0.87	8.74	3.61
0.75	0.00	0.90	7.32	3.85
1.00	0.00	0.90	7.07	4.01

Table 7. Effect of retain ratio on STAMP-LR performance on Llama-3-8B.

Method	$Acc_{f}\!\downarrow$	$Acc_{r}\!\uparrow$	Utility $\downarrow$
GA ( Yao and Xu, 2024 )	100	42.17	6.02
NPO ( Zhang et al., 2024 )	100	48.93	5.45
RMU ( Li et al., 2024a )	100	39.27	6.25
FLAT ( Wang et al., 2024 )	100	51.63	6.05
WGA ( Wang et al., 2025b )	100	44.51	6.13
ASU ( Zade et al., 2026 )	100	53.12	5.48

Table 8. Single-sample unlearning comparison on Llama-3-8B for harmful knowledge removal (

|\mathcal{D}_{f}|=1

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 349 papers

Apr 23, 2026

CAP: Controllable Alignment Prompting for Unlearning in LLMs

Large language models (LLMs) trained on unfiltered corpora inherently risk retaining sensitive information, necessitating selective knowledge unlearning for regulatory compliance and ethical safety. However, existing parameter-modifying methods face fundamental limitations: high computational costs, uncontrollable forgetting boundaries, and strict dependency on model weight access. These constraints render them impractical for closed-source models, yet current non-invasive alternatives remain unsystematic and reliant on empirical experience. To address these challenges, we propose the Controllable Alignment Prompting for Unlearning (CAP) framework, an end-to-end prompt-driven unlearning paradigm. CAP decouples unlearning into a learnable prompt optimization process via reinforcement learning, where a prompt generator collaborates with the LLM to suppress target knowledge while preserving general capabilities selectively. This approach enables reversible knowledge restoration through prompt revocation. Extensive experiments demonstrate that CAP achieves precise, controllable unlearning without updating model parameters, establishing a dynamic alignment mechanism that overcomes the transferability limitations of prior methods.

Zhaokun Wang, Jinyu Guo, Jingwen Pu +7

School of Information and Software Engineering, University of Electronic Science and Technology of China

LLM Alignment LLM Unlearning

Ask PDF

May 16, 2026

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety. Existing machine unlearning methods primarily rely on retraining or aggressive fine-tuning, which are either computationally expensive or prone to degrading related knowledge and overall model utility. In this work, we reformulate machine unlearning as a precise knowledge re-mapping problem via model editing. We propose ZeroUnlearn, a few-shot unlearning framework. It overwrites sensitive inputs by mapping them to a neutral target state and removing their original representations. ZeroUnlearn enforces representational orthogonality through a multiplicative parameter update with a closed-form solution, enabling efficient and targeted unlearning. We further extend ZeroUnlearn to a gradient-based variant for multi-sample unlearning. Experiments demonstrate that our approach outperforms existing baselines while preserving general model utility. Our code is available at the github: https://github.com/XMUDeepLIT/ZeroUnlearn.

Yujie Lin, Chengyi Yang, Zhishang Xiang +2

School of Informatics, Xiamen University, China · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China · School of Film, Xiamen University, China +2

Privacy Leakage in Language Models Model Editing Machine Unlearning LLM Unlearning

Ask PDF

Oct 7, 2026

Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning

Large Language Models (LLMs) inevitably internalize substantial amounts of sensitive or private information during pre-training, while LLM unlearning aims to selectively erase specific knowledge to prevent privacy leakage with minimal loss of model utility. However, existing methods struggle to balance forget quality with utility, and typically incur substantial computational costs due to parameter fine-tuning. To address this, we propose Nullify, a training-free, non-destructive activation steering method for LLM unlearning. Nullify employs steering vectors during inference to redirect privacy-related activations away from their memorized answers, while satisfying a null-space constraint that leaves retained-query activations essentially unaffected to maintain utility. Evaluations on TOFU and MUSE show that Nullify matches or surpasses established baselines in forget quality while achieving near-lossless preservation of model utility. By avoiding weight updates entirely, Nullify serves as an efficient, plug-and-play inference-time intervention framework.

Wei Zhai, Xiang Liu, Qiang Huang +6

College of Computer Science and Artificial Intelligence, Fudan University · BEDI Cloud · School of Electrical and Information Engineering, Tianjin University +3

LLM Unlearning Privacy Leakage in Language Models Activation Steering Causal Interventions in Language Models

Ask PDF

Abstract

Machine Unlearning

Explore similar work

CAP: Controllable Alignment Prompting for Unlearning in LLMs

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning